Related
I'm currently using a keysight VNA product and I control it using PyVisa. Since I have a rapid changing system, I wish to query binary data instead of ascii data from the machine since it is about 10 times faster. The issue I am having is to convert the data to ascii again.
Minimal exampel code:
import pyvisa as visa
import numpy as np
device_adress = ''TCPIP0::localhost::hislip1,4880::INSTR''
rm = visa.ResourceManager('C:\\Windows\\System32\\visa32.dll')
device = rm.open_resource(device_adres)
# presetting device for SNP data measurment
# ...
device.query_ascii_values('CALC:DATA:SNP? 2', container = np.ndarray) # works super but is slow
device.write('FORM:DATA REAL,64')
device.query_binary_values('CALC:DATA:SNP? 2', container = np.ndarray) # 10 times faster but how to read data
Official docs to query binary doesn't give me anything. I found the functions for the code on git here and some helper functions for converting data here, but I am still unable to convert the data such that the converted data is the same as the one I got from the ascii query command. If possible I would like the 'container=np.ndarray' to kept.
Functions from the last link that I have tested:
bin_data = device.query_binary_values('CALC:DATA:SNP? 2', container = np.ndarray)
num = from_binary_block(bin_data) # "Convert a binary block into an iterable of numbers."
ascii_data = to_ascii_block(num) # "Turn an iterable of numbers in an ascii block of data."
but the data from query_ascii_values and the values of ascii_data don't match. Any help is higly appreciated.
Edit:
With the following code
device.write(f"SENS:SWE:POIN 5;")
data_bin = device.query_binary_values('CALC:DATA? SDATA', container=np.ndarray)
I got
data_bin = array([-5.0535379e-34, 1.3452465e-43, -1.7349754e+09, 1.3452465e-43,
-8.6640313e+22, 8.9683102e-44, 5.0314407e-06, 3.1389086e-43,
4.8143607e-36, 3.1389086e-43, -4.1738553e-12, 1.3452465e-43,
-1.5767541e+11, 8.9683102e-44, -2.8241991e+32, 1.7936620e-43,
4.3024710e+16, 1.3452465e-43, 2.1990014e+07, 8.9683102e-44],
dtype=float32)
I am using ipython 5.8.0 on Debian 10.
This is how output looks like:
In [1]: 50*50
Out[1]: 2500
Is it possible to configure ipython to print all numbers with thousands separators? ie:
In [1]: 50*50
Out[1]: 2'500
In [2]: 5000*5000
Out[2]: 25'000'000
And perhaps, is it possible to make ipython also understand thousands separators on input?
In [1]: 5'000*5'000
Out[1]: 25'000'000
UPDATE
The accepted answer from #Chayim Friedman works for integers, but does not work for float:
In [1]: 500.1*500
Out[1]: 250050.0
Also, when it works, it uses , as the character for thousand separator:
In [1]: 500*500
Out[1]: 250,000
Can I use ' instead?
Using ' as thousands separator in input is quite problematic because Python uses ' to delimit strings, but you can use _ (PEP 515, Underscores in Numeric Literals):
Regarding output, this is slightly harder, but can be done using IPython extensions.
Put the following Python code in a new file at ~/.ipython/extensions/thousands_separator.py:
default_int_printer = None
def print_int(number, printer, cycle):
printer.text(f'{number:,}') # You can use `'{:,}'.format(number)` if you're using a Python version older than 3.6
def load_ipython_extension(ipython):
global default_int_printer
default_int_printer = ipython.display_formatter.formatters['text/plain'].for_type(int, print_int)
def unload_ipython_extension(ipython):
ipython.display_formatter.formatters['text/plain'].for_type(int, default_int_printer)
This code tells IPython to replace the default int formatter with one that prints thousand separators when this extension is loaded, and restore the original when it is unloaded.
Edit: If you want a different separator, for instance ', replace the f'{number:,}' with f'{number:,}'.replace(',', "'").
You can load the extension using the magic command %load_ext thousands_separator and unload it using %unload_ext thousands_separator, but if you want it always, you can place it in the default profile.
Run the following code in the terminal:
ipython3 profile create
It will report that a file ~/.ipython/profile_default/ipython_config.py was created. Enter it, and search for the following string:
## A list of dotted module names of IPython extensions to load.
#c.InteractiveShellApp.extensions = []
Replace it with the following:
# A list of dotted module names of IPython extensions to load.
c.InteractiveShellApp.extensions = [
'thousands_separator'
]
This tells IPython to load this extension by default.
Done!
Edit: I saw that you want to a) use ' as separator, and b) do the same for floats:
Using different separator is quite easy: just str.replace():
def print_int(number, printer, cycle):
printer.text(f'{number:,}'.replace(',', "'"))
Doing the same for floats is also easy: just setup print_int so it prints floats to. I also suggest to change the name to print_number.
Final code:
default_int_printer = None
default_float_printer = None
def print_number(number, printer, cycle):
printer.text(f'{number:,}'.replace(',', "'"))
def load_ipython_extension(ipython):
global default_int_printer
global default_float_printer
default_int_printer = ipython.display_formatter.formatters['text/plain'].for_type(int, print_number)
default_float_printer = ipython.display_formatter.formatters['text/plain'].for_type(float, print_number)
def unload_ipython_extension(ipython):
ipython.display_formatter.formatters['text/plain'].for_type(int, default_int_printer)
ipython.display_formatter.formatters['text/plain'].for_type(float, default_float_printer)
After update: you can subclass int:
class Int(int):
def __repr__(self):
return "{:,}".format(self)
Int(1000)
# 1,000
I don't believe you can achieve all that you are looking for without rewriting the iPython interpreter, which means changing the Python language specification, to be able to input numbers with embedded ' characters and have them ignored. But you can achieve some of it. Subclassing the int class is a good start. But you should also overload the various operators you plan on using. For example:
class Integer(int):
def __str__(self):
# if you want ' as the separator:
return "{:,}".format(self).replace(",", "'")
def __add__(self, x):
return Integer(int(self) + x)
def __mul__(self, x):
return Integer(int(self) * x)
"""
define other operations: __sub__, __floordiv__, __mod__, __neg__, etc.
"""
i1 = Integer(2)
i2 = Integer(1000) + 4.5 * i1
print(i2)
print(i1 * (3 + i2))
Prints:
1'009
2'024
Update
It seems that for Python 3.7 you need to override the __str__ method rather than the __repr__ method. This works for Python 3.8 and should work for later releases as well.
Update 2
import locale
#locale.setlocale(locale.LC_ALL, '') # probably not required
print(locale.format_string("%d", 1255000, grouping=True).replace(",", "'"))
Prints:
1'255'000
An alternative if you have package Babel from the PyPi repository:
from babel import Locale
from babel.numbers import format_number
locale = Locale('en', 'US')
locale.number_symbols['group'] = "'"
print(format_number(1255000, locale='en_US'))
Prints:
1'255'000
Or if you prefer to custom-tailor a locale just for this purpose and leave the standard en_US locale unmodified. This also shows how you can parse input values:
from copy import deepcopy
from babel import Locale
from babel.numbers import format_number, parse_number
my_locale = deepcopy(Locale('en', 'US'))
my_locale.number_symbols['group'] = "'"
print(format_number(1255000, locale=my_locale))
print(parse_number("1'125'000", locale=my_locale))
Prints:
1'255'000
1125000
Based on PEP-0378, you can use the following code:
a = 1200
b = 500
c = 10
#res = a
#res = a*b
res = a*b*c
dig = len(str(res)) # to figure out how many digits are required in result
print(format(res, "{},d".format(dig)))
It will produce:
6,000,000
I want to use f-string with my string variable, not with string defined with a string literal, "...".
Here is my code:
name=["deep","mahesh","nirbhay"]
user_input = r"certi_{element}" # this string I ask from user
for element in name:
print(f"{user_input}")
This code gives output:
certi_{element}
certi_{element}
certi_{element}
But I want:
certi_{deep}
certi_{mahesh}
certi_{nirbhay}
How can I do this?
f"..." strings are great when interpolating expression results into a literal, but you don't have a literal, you have a template string in a separate variable.
You can use str.format() to apply values to that template:
name=["deep","mahesh","nirbhay"]
user_input = "certi_{element}" # this string i ask from user
for value in name:
print(user_input.format(element=value))
String formatting placeholders that use names (such as {element}) are not variables. You assign a value for each name in the keyword arguments of the str.format() call instead. In the above example, element=value passes in the value of the value variable to fill in the placeholder with the element.
Unlike f-strings, the {...} placeholders are not expressions and you can't use arbitrary Python expressions in the template. This is a good thing, you wouldn't want end-users to be able to execute arbitrary Python code in your program. See the Format String Syntax documenation for details.
You can pass in any number of names; the string template doesn't have to use any of them. If you combine str.format() with the **mapping call convention, you can use any dictionary as the source of values:
template_values = {
'name': 'Ford Prefect',
'number': 42,
'company': 'Sirius Cybernetics Corporation',
'element': 'Improbability Drive',
}
print(user_input.format(**template_values)
The above would let a user use any of the names in template_values in their template, any number of times they like.
While you can use locals() and globals() to produce dictionaries mapping variable names to values, I'd not recommend that approach. Use a dedicated namespace like the above to limit what names are available, and document those names for your end-users.
If you define:
def fstr(template):
return eval(f"f'{template}'")
Then you can do:
name=["deep","mahesh","nirbhay"]
user_input = r"certi_{element}" # this string i ask from user
for element in name:
print(fstr(user_input))
Which gives as output:
certi_deep
certi_mahesh
certi_nirbhay
But be aware that users can use expressions in the template, like e.g.:
import os # assume you have used os somewhere
user_input = r"certi_{os.environ}"
for element in name:
print(fstr(user_input))
You definitely don't want this!
Therefore, a much safer option is to define:
def fstr(template, **kwargs):
return eval(f"f'{template}'", kwargs)
Arbitrary code is no longer possible, but users can still use string expressions like:
user_input = r"certi_{element.upper()*2}"
for element in name:
print(fstr(user_input, element=element))
Gives as output:
certi_DEEPDEEP
certi_MAHESHMAHESH
certi_NIRBHAYNIRBHAY
Which may be desired in some cases.
If you want the user to have access to your namespace, you can do that, but the consequences are entirely on you. Instead of using f-strings, you can use the format method to interpolate dynamically, with a very similar syntax.
If you want the user to have access to only a small number of specific variables, you can do something like
name=["deep", "mahesh", "nirbhay"]
user_input = "certi_{element}" # this string i ask from user
for element in name:
my_str = user_input.format(element=element)
print(f"{my_str}")
You can of course rename the key that the user inputs vs the variable name that you use:
my_str = user_input.format(element=some_other_variable)
And you can just go and let the user have access to your whole namespace (or at least most of it). Please don't do this, but be aware that you can:
my_str = user_input.format(**locals(), **globals())
The reason that I went with print(f'{my_str}') instead of print(my_str) is to avoid the situation where literal braces get treated as further, erroneous expansions. For example, user_input = 'certi_{{{element}}}'
I was looking for something similar with your problem.
I came across this other question's answer: https://stackoverflow.com/a/54780825/7381826
Using that idea, I tweaked your code:
user_input = r"certi_"
for element in name:
print(f"{user_input}{element}")
And I got this result:
certi_deep
certi_mahesh
certi_nirbhay
If you would rather stick to the layout in the question, then this final edit did the trick:
for element in name:
print(f"{user_input}" "{" f"{element}" "}")
Reading the security concerns of all other questions, I don't think this alternative has serious security risks because it does not define a new function with eval().
I am no security expert so please do correct me if I am wrong.
This is what you’re looking for. Just change the last line of your original code:
name=["deep","mahesh","nirbhay"]
user_input = "certi_{element}" # this string I ask from user
for element in name:
print(eval("f'" + f"{user_input}" + "'"))
When using ruamel.yaml version 0.15.92 with Python 3.6.6 on CentOS 7, I cannot seem to update the value of an anchored scalar in a sequence without destroying the anchor itself or creating invalid YAML from the next dump.
I have attempted to recreate the original node type with the new value (old PlainScalarString -> new PlainScalarString, old FoldedScalarString -> new FoldedScalarString, etc), copying the anchor to it. While this restores the anchor to the updated scalar value, it also creates invalid YAML because the first alias later in the YAML file duplicates the same anchor name and assigns to it the old value of the scalar I'm trying to update.
I then attempted to replace all of the affected aliases with actual alias text -- like *anchor_name -- but that causes the value to become quoted like '*anchor_name', rendering the alias useless.
I reverted that and then attempted to suppress the duplicate anchor name (by setting always_dump=False on every affected alias). While that does suppress the duplicate anchor name, it unfortunately just dumps the old value of the anchored scalar.
My entire test data is as follows; assume this is named test.yaml:
# Header comment
---
# Post-header comment
# Reusable aliases
aliases:
- &plain_value This is unencrypted
- &string_password ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAYnFbMveZGBgd9aw7h4VV+M202zRdcP96UQs1q+ViznJK2Ee08hoW9jdIqVhNaecYALUihKjVYijJa649VF7BLZXV0svLEHD8LZeduoLS3iC9uszdhDFB2Q6R/Vv/ARjHNoWc6/D0nFN9vwcrQNITnvREl0WXYpR9SmW0krUpyr90gSAxTxPNJVlEOtA0afeJiXOtQEu/b8n+UDM3eXXRO+2SEXM4ub7fNcj6V9DgT3WwKBUjqzQ5DicnB19FNQ1cBGcmCo8qRv0JtbVqZ4+WJFGc06hOTcAJPsAaWWUn80ChcTnl4ELNzpJFoxAxHgepirskuIvuWZv3h/PL8Ez3NDBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBSuVIsvWXMmdFJtJmtJxXxgCAGFCioe/zdphGqynmj6vVDnCjA3Xc0VPOCmmCl/cTKdg==]
- &block_password >
ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
DQYJKoZIhvcNAQEBBQAEggEAojErrxuNcdX6oR+VA/I3PyuV2CwXx166nIUp
asEHo1/CiCIoE3qCnjK2FJF8vg+l3AqRmdb7vYrqQ+30RFfHSlB9zApSw8NW
tnEpawX4hhKAxnTc/JKStLLu2k7iZkhkor/UA2HeVJcCzEeYAwuOQRPaolmQ
TGHjvm2w6lhFDKFkmETD/tq4gQNcOgLmJ+Pqhogr/5FmGOpJ7VGjpeUwLteM
er3oQozp4l2bUTJ8wk9xY6cN+eeOIcWXCPPdNetoKcVropiwrYH8QV4CZ2Ky
u0vpiybEuBCKhr1EpfqhrtuG5s817eOb7+Wf5ctR0rPuxlTUqdnDY31zZ3Kb
mcjqHDBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBATq6BjaxU2bfcLL5S
bxzsgCDsWzggzxsCw4Dp0uYLwvMKjJEpMLeFXGrLHJzTF6U2Nw==]
top_key: unencrypted value
top_alias: *plain_value
top::hash:
ignore: more
# This pulls its string-form value from above
stringified_alias: *string_password
sub:
ignore: value
key: unencrypted subbed-value
# This pulls its block-form value from above
blocked_alias: *block_password
sub_more:
# This is a stringified EYAML value, NOT an alias
inline_string: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAafmyrrae2kx8HdyPmn/RHQRcTPhqpx5Idm12hCDCIbwVM++H+c620z4EN2wlugz/GcLaiGsybaVWzAZ+3r+1+EwXn5ec4dJ5TTqo7oxThwUMa+SHliipDJwGoGii/H+y2I+3+irhDYmACL2nyJ4dv4IUXwqkv6nh1J9MwcOkGES2SKiDm/WwfkbPIZc3ccp1FI9AX/m3SVqEcvsrAfw6HtkolM22csfuJREHkTp7nBapDvOkWn4plzfOw9VhPKhq1x9DUCVFqqG/HAKv++v4osClK6k1MmSJWaMHrW1z3n7LftV9ZZ60E0Cgro2xSaD+itRwBp07H0GeWuoKB4+44TBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCRv9r2lvQ1GJMoD064EtdigCCw43EAKZWOc41yEjknjRaWDm1VUug6I90lxCsUrxoaMA==]
# Also NOT an alias, in block form
block_string: >
ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
DQYJKoZIhvcNAQEBBQAEggEAafmyrrae2kx8HdyPmn/RHQRcTPhqpx5Idm12
hCDCIbwVM++H+c620z4EN2wlugz/GcLaiGsybaVWzAZ+3r+1+EwXn5ec4dJ5
TTqo7oxThwUMa+SHliipDJwGoGii/H+y2I+3+irhDYmACL2nyJ4dv4IUXwqk
v6nh1J9MwcOkGES2SKiDm/WwfkbPIZc3ccp1FI9AX/m3SVqEcvsrAfw6Htko
lM22csfuJREHkTp7nBapDvOkWn4plzfOw9VhPKhq1x9DUCVFqqG/HAKv++v4
osClK6k1MmSJWaMHrW1z3n7LftV9ZZ60E0Cgro2xSaD+itRwBp07H0GeWuoK
B4+44TBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCRv9r2lvQ1GJMoD064
EtdigCCw43EAKZWOc41yEjknjRaWDm1VUug6I90lxCsUrxoaMA==]
# Signature line
There are two forms of this issue, so here are two code examples for reproducing the conditions:
First, "How can we most simply update the value of an anchored scalar in a sequence without destroying the anchor or its aliases?" This looks like:
with open('test.yaml', 'r') as f:
yaml_data = yaml.load(f)
yaml_data['aliases'][1] = "New string password"
yaml.dump(yaml_data, sys.stdout)
Note that this destroys the anchor. I would very much prefer the solution look as similar to this first snippet as possible; perhaps something like yaml_data['aliases'][1].set_value("New string password") # Changes only the scalar value while preserving the original anchor, comments, position, et al..
Second, "If we must instead wrap the new value in some object to preserve the anchor (and other attributes of the entry being replaced), what is the simplest approach which also preserves all aliases that refer to it (such that they adopt the updated value) when dumped?" My attempt to solve this requires quite a lot more code including recursive functions. Since SO guidelines advise against dumping large code, I will offer the relevant bits. Please assume the unlisted code is working perfectly well.
### <snip def FindEYAMLPaths(...) returns lists of paths through the YAML to every value starting with 'ENC['>
### <snip def GetYAMLValue(...) returns the node -- as a PlainScalarString, FoldedScalarString, et al. -- identified by a path from FindEYAMLPaths>
### <snip def DisableAnchorDump(...) sets `anchor.always_dump=False` if the node has an anchor attribute>
def ReplaceYAMLValue(value, data, path=None):
if path is None:
return
ref = data
last_ref = path.pop()
for p in path:
ref = ref[p]
# All I'm trying to do here is change the scalar value without disrupting its comments, anchor, positioning, or any of its aliases.
# This succeeds in changing the scalar value and preserving its original anchor, but disrupts its aliases which insist on preserving the old value.
if isinstance(ref[last_ref], PlainScalarString):
ref[last_ref] = PlainScalarString(value, anchor=ref[last_ref].anchor.value)
elif isinstance(ref[last_ref], FoldedScalarString):
ref[last_ref] = FoldedScalarString(value, anchor=ref[last_ref].anchor.value)
else:
ref[last_ref] = value
with open('test.yaml', 'r') as f:
yaml_data = yaml.load(f)
seen_anchors = []
for path in FindEYAMLPaths(yaml_data):
if path is None:
continue
node = GetYAMLValue(yaml_data, deque(path))
if hasattr(node, 'anchor'):
test_anchor = node.anchor.value
if test_anchor is not None:
if test_anchor in seen_anchors:
# This is expected to just be an alias, pointing at the newly updated anchor
DisableAnchorDump(node)
continue
seen_anchors.append(test_anchor)
ReplaceYAMLValue("New string password", yaml_data, path)
yaml.dump(yaml_data, sys.stdout)
Note that this produces valid YAML except that all of the affected aliases are gone, replaced instead by the old value of the anchored scalar.
I expect to be able to change the value of an aliased scalar in a sequence without disrupting any other part of the YAML content. Based on other posts I've seen about ruamel.yaml, I fully accept that I may need to dump the updated YAML to file and reload it for the in-memory aliases to update to the new value. I simply expect to change:
Input File
aliases:
- &some_anchor Old value
usage: *some_anchor
to:
Output File
aliases:
- &some_anchor NEW VALUE
usage: *some_anchor
Instead, here's the output from the above two examples:
First, notice that the original anchor was destroyed and the value for top::hash:stringified_alias: now carries the original anchor and old value instead of the alias to the newly updated scalar value at ['aliases'][1]:
---
# Post-header comment
# Reusable aliases
aliases:
- &plain_value This is unencrypted
- New string password
- &block_password >
ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
DQYJKoZIhvcNAQEBBQAEggEAojErrxuNcdX6oR+VA/I3PyuV2CwXx166nIUp
asEHo1/CiCIoE3qCnjK2FJF8vg+l3AqRmdb7vYrqQ+30RFfHSlB9zApSw8NW
tnEpawX4hhKAxnTc/JKStLLu2k7iZkhkor/UA2HeVJcCzEeYAwuOQRPaolmQ
TGHjvm2w6lhFDKFkmETD/tq4gQNcOgLmJ+Pqhogr/5FmGOpJ7VGjpeUwLteM
er3oQozp4l2bUTJ8wk9xY6cN+eeOIcWXCPPdNetoKcVropiwrYH8QV4CZ2Ky
u0vpiybEuBCKhr1EpfqhrtuG5s817eOb7+Wf5ctR0rPuxlTUqdnDY31zZ3Kb
mcjqHDBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBATq6BjaxU2bfcLL5S
bxzsgCDsWzggzxsCw4Dp0uYLwvMKjJEpMLeFXGrLHJzTF6U2Nw==]
# ... snip ...
top::hash:
ignore: more
# This pulls its string-form value from above
stringified_alias: &string_password ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAYnFbMveZGBgd9aw7h4VV+M202zRdcP96UQs1q+ViznJK2Ee08hoW9jdIqVhNaecYALUihKjVYijJa649VF7BLZXV0svLEHD8LZeduoLS3iC9uszdhDFB2Q6R/Vv/ARjHNoWc6/D0nFN9vwcrQNITnvREl0WXYpR9SmW0krUpyr90gSAxTxPNJVlEOtA0afeJiXOtQEu/b8n+UDM3eXXRO+2SEXM4ub7fNcj6V9DgT3WwKBUjqzQ5DicnB19FNQ1cBGcmCo8qRv0JtbVqZ4+WJFGc06hOTcAJPsAaWWUn80ChcTnl4ELNzpJFoxAxHgepirskuIvuWZv3h/PL8Ez3NDBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBSuVIsvWXMmdFJtJmtJxXxgCAGFCioe/zdphGqynmj6vVDnCjA3Xc0VPOCmmCl/cTKdg==]
# ... snip ...
Second, notice that ['aliases'][1] now looks correct -- it is the new value with the original anchor -- but where I expect to see aliases to it, I instead see the old value. I expect to see *string_password instead of ENC[...].
---
# Post-header comment
# Reusable aliases
aliases:
- &plain_value This is unencrypted
- &string_password New string password
- &block_password >-
New string password
# ... snip ...
top::hash:
ignore: more
# This pulls its string-form value from above
stringified_alias: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAYnFbMveZGBgd9aw7h4VV+M202zRdcP96UQs1q+ViznJK2Ee08hoW9jdIqVhNaecYALUihKjVYijJa649VF7BLZXV0svLEHD8LZeduoLS3iC9uszdhDFB2Q6R/Vv/ARjHNoWc6/D0nFN9vwcrQNITnvREl0WXYpR9SmW0krUpyr90gSAxTxPNJVlEOtA0afeJiXOtQEu/b8n+UDM3eXXRO+2SEXM4ub7fNcj6V9DgT3WwKBUjqzQ5DicnB19FNQ1cBGcmCo8qRv0JtbVqZ4+WJFGc06hOTcAJPsAaWWUn80ChcTnl4ELNzpJFoxAxHgepirskuIvuWZv3h/PL8Ez3NDBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBSuVIsvWXMmdFJtJmtJxXxgCAGFCioe/zdphGqynmj6vVDnCjA3Xc0VPOCmmCl/cTKdg==]
# ... snip ...
If you read in an anchored scalar, like your This is unencrypted,
using ruamel.yaml, you get a PlainScalarString object (or one of the other ScalarString
subclasses), which is an extremely thin layer around the basic string
type. That layer has an attribute to store an anchor if applicable (other uses are primarily to
maintain quoting/literal/folding style information). And any aliases using that anchor refer to the same ScalarString instance.
When dumping the anchor attribute is not used to create aliases, that
is is done in the normal way by having multiple references to the same
object. The attribute is only used to write the anchor id and also
does so if there is an attribute but no further references (i.e. an anchor without aliases).
So it is not surprising that if you replace such an object with
multiple references (either at the anchor spot or any of the alias
spots) that the reference disappears. If you then also force the same
anchor name on some other object, you get duplicate anchors, contrary
to the normal anchor/alias generation there is no check done on
"forced" anchors.
Since the ScalarString is such a thin wrapper, they are essentially
immutable objects, just like the string itself. Unlike with aliased
dicts and lists which are collection objects that can be emptied and
then filled (instead of replaced by a new instance), you cannot do
that with string.
The implementation of ScalarString can of course be changed, so you
can have your set_values() method, but involves creating alternative
classes for all the objects (PlainScalarString,
FoldedScalarString). You would have to make sure
these get used for constructing and for representing and then
preferable also behave like normal strings as far as you need it, so
at least you can print.
That is relatively easy to do but requires copying and slightly modifyging several
tens of lines of code
I think it is easier to leave the ScalarStrings in place as is (i.e
being immutable) and do what you need to do if you want to change all
occurences (i.e. references): update all the references to the
original. If your datastructure would contain millions of nodes that
might be prohibitively time consuming, but still would be afraction of what
loading and dumping the YAML itself would take:
import sys
from pathlib import Path
import ruamel.yaml
in_file = Path('test.yaml')
def update_aliased_scalar(data, obj, val):
def recurse(d, ref, nv):
if isinstance(d, dict):
for i, k in [(idx, key) for idx, key in enumerate(d.keys()) if key is ref]:
d.insert(i, nv, d.pop(k))
for k, v in d.non_merged_items():
if v is ref:
d[k] = nv
else:
recurse(v, ref, nv)
elif isinstance(d, list):
for idx, item in enumerate(d):
if item is ref:
d[idx] = nv
else:
recurse(item, ref, nv)
if hasattr(obj, 'anchor'):
recurse(data, obj, type(obj)(val, anchor=obj.anchor.value))
else:
recurse(data, obj, type(obj)(val))
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=2, sequence=4, offset=2)
yaml.preserve_quotes = True
data = yaml.load(in_file)
update_aliased_scalar(data, data['aliases'][1], "New string password")
update_aliased_scalar(data, data['top::hash']['sub']['blocked_alias'], "New block password\n")
yaml.dump(data, sys.stdout)
which gives:
# Post-header comment
# Reusable aliases
aliases:
- &plain_value This is unencrypted
- &string_password New string password
- &block_password >
New block password
top_key: unencrypted value
top_alias: *plain_value
top::hash:
ignore: more
# This pulls its string-form value from above
stringified_alias: *string_password
sub:
ignore: value
key: unencrypted subbed-value
# This pulls its block-form value from above
blocked_alias: *block_password
sub_more:
# This is a stringified EYAML value, NOT an alias
inline_string: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAafmyrrae2kx8HdyPmn/RHQRcTPhqpx5Idm12hCDCIbwVM++H+c620z4EN2wlugz/GcLaiGsybaVWzAZ+3r+1+EwXn5ec4dJ5TTqo7oxThwUMa+SHliipDJwGoGii/H+y2I+3+irhDYmACL2nyJ4dv4IUXwqkv6nh1J9MwcOkGES2SKiDm/WwfkbPIZc3ccp1FI9AX/m3SVqEcvsrAfw6HtkolM22csfuJREHkTp7nBapDvOkWn4plzfOw9VhPKhq1x9DUCVFqqG/HAKv++v4osClK6k1MmSJWaMHrW1z3n7LftV9ZZ60E0Cgro2xSaD+itRwBp07H0GeWuoKB4+44TBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCRv9r2lvQ1GJMoD064EtdigCCw43EAKZWOc41yEjknjRaWDm1VUug6I90lxCsUrxoaMA==]
# Also NOT an alias, in block form
block_string: >
ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
DQYJKoZIhvcNAQEBBQAEggEAafmyrrae2kx8HdyPmn/RHQRcTPhqpx5Idm12
hCDCIbwVM++H+c620z4EN2wlugz/GcLaiGsybaVWzAZ+3r+1+EwXn5ec4dJ5
TTqo7oxThwUMa+SHliipDJwGoGii/H+y2I+3+irhDYmACL2nyJ4dv4IUXwqk
v6nh1J9MwcOkGES2SKiDm/WwfkbPIZc3ccp1FI9AX/m3SVqEcvsrAfw6Htko
lM22csfuJREHkTp7nBapDvOkWn4plzfOw9VhPKhq1x9DUCVFqqG/HAKv++v4
osClK6k1MmSJWaMHrW1z3n7LftV9ZZ60E0Cgro2xSaD+itRwBp07H0GeWuoK
B4+44TBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCRv9r2lvQ1GJMoD064
EtdigCCw43EAKZWOc41yEjknjRaWDm1VUug6I90lxCsUrxoaMA==]
# Signature line
As you can see the anchors are preserved and it doesn't matter for update_aliased_scalar if you
provide the anchored "place" or one of the aliased places as a reference.
The above recurse also handles keys that are aliased, as it is perfectly fine for a key in a YAML mapping to have an anchor or to be an alias. You can even have an anchored key with a value that is an alias to the corresponding key.
It would be very nice to have support for in-place modification of existing anchored fields with types ScalarFloat/ScalarInt etc. YAML is often used for config files. One common use case I encountered is to create multiple config files from a very large template config file with only small changes made to the new files. I would load the template file into CommentedMap, modify a small set of keys in place and dump it back into a new yaml config file. This flow works very nicely if the keys to be changed are not anchored. When they are anchored, the anchors are duplicated in the new files as reported by OP and render them invalid. Manually addressing each anchored key in post-processing can be daunting when there are a large number of them.
I have a file with the following data:
classes:
- 9:00
- 10:20
- 12:10
(and so on up to 21:00)
I use python3 and yaml module to parse it. Precisely, the source is config = yaml.load (open (filename, 'r')). But then, when I print config, I get the following output for this part of data:
'classes': [540, 630, 730, 820, 910, 1000, 1090, 1180],
The values in the list are ints.
While previously, when I used python2 (and BaseLoader for YAML), I got the values as strings, and I use them as such. BaseLoader is now not acceptable since I want to read unicode strings from file, and it gives me byte-strings.
So, first, why pyyaml does parse my data as ints?
And, second, how do I prevent pyyaml from doing this? Is it possible to do that without changing data file (e.g. without adding !!str)?
The documentation of YAML is a bit difficult to "parse" so I can imagine you missed this little bit of info about colons:
Normally, YAML insists the “:” mapping value indicator be separated from the value by white space. A benefit of this restriction is that the “:” character can be used inside plain scalars, as long as it is not followed by white space. This allows for unquoted URLs and timestamps. It is also a potential source for confusion as “a:1” is a plain scalar and not a key: value pair.
And what you have there in your input is a sexagesimal and your 9:00 is considered to be similar to 9 minutes and 0 seconds, equalling a total of 540 seconds.
Unfortunately this doesn't get constructed as some special Sexagesimal instance that can be used for calculations as if it were an integer but can be printed in its original form. Therefore, if you want to use this as a string internally you have to single quote them:
classes:
- '9:00'
- '10:20'
- '12:10'
which is what you would get if you dump {'classes': ['9:00', '10:20', '12:10']} (and note that the unambiguous classes doesn't get any quotes).
That the BaseLoader gives you strings is not surprising. The BaseConstructor that is used by the BaseLoader handles any scalar as string, including integers, booleans and "your" sexagesimals:
import ruamel.yaml as yaml
yaml_str = """\
classes:
- 12345
- 10:20
- abc
- True
"""
data = yaml.load(yaml_str, Loader=yaml.BaseLoader)
print(data)
data = yaml.load(yaml_str, Loader=yaml.SafeLoader)
gives:
{u'classes': [u'12345', u'10:20', u'abc', u'True']}
{'classes': [12345, 620, 'abc', True]}
If you really don't want to use quotes, then you have to "reset" the implicit resolver for scalars that start with numbers:
import ruamel.yaml as yaml
from ruamel.yaml.resolver import Resolver
import re
yaml_str = """\
classes:
- 9:00
- 10:20
- 12:10
"""
for ch in list(u'-+0123456789'):
del Resolver.yaml_implicit_resolvers[ch]
Resolver.add_implicit_resolver(
u'tag:yaml.org,2002:int',
re.compile(u'''^(?:[-+]?0b[0-1_]+
|[-+]?0o?[0-7_]+
|[-+]?(?:0|[1-9][0-9_]*)
|[-+]?0x[0-9a-fA-F_]+)$''', re.X), # <- copy from resolver.py without sexagesimal support
list(u'-+0123456789'))
data = yaml.load(yaml_str, Loader=yaml.SafeLoader)
print(data)
gives you:
{'classes': ['9:00', '10:20', '12:10']}
You should probably check the documentation of YAML
The colon are for mapping values.
I presume you want a string and not an integer, so you should double quote your strings.