I have run out of disk space on my cluster machine. Doing
du -h --max-depth=20 | sort -hr > size_of_folders.txt
reveals that there is a directory called ./miniconda3/pkgs that is taking up 22Gbs.
Looking in this folder at the subfolders, I don't know what most of these . For example here are the top 50 entries:
804M ./qt-main-5.15.4-ha5833f6_2
769M ./qt-5.12.9-h1304e3e_6
629M ./qt-5.9.7-h5867ecd_1
629M ./cache
619M ./qt-5.9.7-h0c104cb_3
482M ./mkl-2020.2-256
481M ./mkl-2020.2-256/lib
360M ./qt-main-5.15.4-ha5833f6_2/include
359M ./qt-main-5.15.4-ha5833f6_2/include/qt
343M ./qt-5.12.9-h1304e3e_6/include/qt
343M ./qt-5.12.9-h1304e3e_6/include
321M ./qt-5.9.7-h5867ecd_1/include
321M ./qt-5.9.7-h0c104cb_3/include
320M ./qt-5.9.7-h5867ecd_1/include/qt
320M ./qt-5.9.7-h0c104cb_3/include/qt
307M ./libdb-6.2.32-h9c3ff4c_0
299M ./libdb-6.2.32-h9c3ff4c_0/docs
244M ./scipy-1.7.3-py310hea5193d_0
243M ./scipy-1.7.3-py310hea5193d_0/lib/python3.10
243M ./scipy-1.7.3-py310hea5193d_0/lib
242M ./scipy-1.7.3-py310hea5193d_0/lib/python3.10/site-packages/scipy
242M ./scipy-1.7.3-py310hea5193d_0/lib/python3.10/site-packages
221M ./pandas-1.3.4-py310hb5077e9_1
220M ./pandas-1.3.4-py310hb5077e9_1/lib/python3.10/site-packages
220M ./pandas-1.3.4-py310hb5077e9_1/lib/python3.10
220M ./pandas-1.3.4-py310hb5077e9_1/lib
219M ./python-3.8.3-cpython_he5300dc_0
219M ./pandas-1.3.4-py310hb5077e9_1/lib/python3.10/site-packages/pandas
199M ./python-3.8.3-cpython_he5300dc_0/lib
196M ./python-3.8.3-hcff3b4d_0
175M ./python-3.8.3-hcff3b4d_0/lib
172M ./pandas-1.4.1-py310hb5077e9_0
171M ./pandas-1.4.1-py310hb5077e9_0/lib/python3.10/site-packages
171M ./pandas-1.4.1-py310hb5077e9_0/lib/python3.10
171M ./pandas-1.4.1-py310hb5077e9_0/lib
170M ./pandas-1.4.3-py310h769672d_0
170M ./pandas-1.4.2-py310h769672d_2
170M ./pandas-1.4.1-py310hb5077e9_0/lib/python3.10/site-packages/pandas
169M ./pandas-1.4.3-py310h769672d_0/lib/python3.10/site-packages
169M ./pandas-1.4.3-py310h769672d_0/lib/python3.10
169M ./pandas-1.4.3-py310h769672d_0/lib
169M ./pandas-1.4.2-py310h769672d_2/lib/python3.10/site-packages
169M ./pandas-1.4.2-py310h769672d_2/lib/python3.10
169M ./pandas-1.4.2-py310h769672d_2/lib
168M ./perl-5.32.1-1_h7f98852_perl5
168M ./pandas-1.4.3-py310h769672d_0/lib/python3.10/site-packages/pandas
168M ./pandas-1.4.2-py310h769672d_2/lib/python3.10/site-packages/pandas
166M ./statsmodels-0.13.1-py310h96516ba_0
165M ./statsmodels-0.13.1-py310h96516ba_0/lib/python3.10/site-packages 165M ./statsmodels-0.13.1-py310h96516ba_0/lib/python3.10`
Can someone explain if these are deletable? These recognisable packages - such as pandas and scipy - have strange suffixes and I'm not sure if they're deletable.
I presume I can delete the cache but I'm not sure about the rest.
Related
I'm having trouble getting an SVG properly rendered into aframe's scene. I went the htmltexture-component route, which has a dependency on the draw-component.
The problem is in getting the SVG's background to render as transparent. It simply is not working. Instead I'm given a default white background drawn on the canvas. My SVG was created in Illustrator and is a valid HTML object with no background (as a color or a raster image).
<div id="bears">
<svg xmlns="http://www.w3.org/2000/svg" width="518" height="500" xmlns:xlink="http://www.w3.org/1999/xlink">
<path d="M168.958 480.023c-12.18-1.57-24.62-6.847-34.293-14.544-9.103-7.25-11.733-15.34-7.8-23.99 4.594-10.11 22.386-19.1 27.564-13.92 1.71 1.71 2.03 5.14.68 7.27-.63.99-1.03.66-1.51-1.25-1.37-5.43-13.5-2.54-16.44 3.92-2.72 5.95-.57 14.06 4.3 16.28 5.39 2.45 15.09 1.04 23.89-3.49 3.63-1.87 13.35-9.11 19.14-14.25 12.62-11.23 19.38-16.07 28.49-20.42 5.93-2.84 7.37-3.11 16.5-3.11 8.3 0 10.88.39 15.18 2.32 2.84 1.27 5.95 3.17 6.9 4.22 2.05 2.26 3.74 2.42 5.34.5 2.24-2.71 11.83-6.6 17.78-7.23 6.83-.72 17.37.833 22.4 3.3 3.87 1.892 14 8.59 17.72 11.72 1.38 1.155 6.14 5.31 10.59 9.225 14.51 12.8 25.37 18.42 35.56 18.42 4.85 0 9.52-2.153 11.11-5.117 1.75-3.28 1.46-10.15-.56-13.24-1.83-2.793-7.96-6.428-10.258-6.08-.682.102-2.15.243-3.25.312-1.55.097-2 .845-2.03 3.375-.03 3.073-.11 3.144-1.5 1.31-3.32-4.39-.46-9.56 5.28-9.56 7.277 0 18.478 7.596 22.327 15.14 2.34 4.59 2.48 10.893.34 15.403-3.748 7.9-16.18 16.29-31.223 21.08-13.8 4.39-45.63 4.12-65.69-.56-11.47-2.67-21.36-7.31-27.05-12.67-3.21-3.02-6.45-3.65-6.45-1.26 0 .51-2.65 2.63-5.88 4.72-10.35 6.7-24.14 10.68-42.05 12.17-12.99 1.08-26.83 1.08-35.11.02zM374.5 360.566c-7.77-1.094-14.833-3.225-33.893-10.225-2.14-.78-9.34-4.27-16-7.75-14.11-7.37-19.35-9.6-29.607-12.57-12.336-3.57-13.295-3.8-18.5-4.48-8.943-1.16-8.8-.78-6.03-16.03.2-1.1.4-3.03.447-4.3.08-2.21.308-2.27 6.333-1.66 6.334.65 20.588 3.89 24.25 5.51 1.1.49 4.505 1.97 7.567 3.29 3.06 1.32 7.894 4.04 10.74 6.04 2.845 2 5.515 3.64 5.933 3.64.418 0 1.99 1.02 3.49 2.25 4.238 3.49 7.614 5.83 16.27 11.27 7.074 4.45 9.82 5.9 18 9.5 4.746 2.1 11.452 3.81 17.5 4.48 3.575.4 7.85.88 9.5 1.08 3.93.47 14.805-2.08 18.27-4.28 7.812-4.96 12.43-11.71 14.7-21.49 2.514-10.84-.142-21.28-6.64-26.082-5.234-3.87-16.747-1.88-24.316 4.2-1.093.88-.412-.367 1.512-2.76 8.517-10.61 20.494-11.775 30.528-2.965 7.336 6.44 10.342 17.7 7.88 29.525-1.44 6.92-6.636 18.58-9.84 22.07-10.297 11.237-25.51 14.96-48.094 11.78zm-262-2.216c-6.116-.74-13.637-3.196-17-5.55-5.537-3.875-9.688-8.793-13.294-15.754-9.527-18.388-7.015-35.84 6.445-44.778 8.89-5.902 16.55-4.74 25.97 3.944 2.41 2.224 4.38 4.548 4.37 5.165-.01.818-.21.822-.75.014-1.13-1.7-10.33-6.31-13.97-7-15.47-2.9-24.2 15.75-15.88 33.94 8 17.5 25.7 23.2 50.15 16.16 11.14-3.2 19.74-7.56 34.28-17.36 6.22-4.19 12.07-8.07 13-8.61.93-.55 3.04-1.93 4.69-3.08 8.9-6.17 22.62-11.34 35.5-13.39 4.13-.65 8.51-1.35 9.74-1.54 1.23-.19 2.52.11 2.87.68.35.56 1.16 5.43 1.81 10.82l1.17 9.8-5.03.6c-5.65.666-19.46 3.85-27.04 6.237-2.75.864-13.55 5.76-24 10.884-22.3 10.93-26.66 12.68-40 16.04-9.48 2.39-25.33 3.744-33 2.82zM174 237.527c-23.41-4.077-43.362-18.19-56.588-40.027-2.58-4.258-7.44-15.748-8.04-19-1.02-5.543-2.358-14.3-2.364-15.487-.004-.817 1.522.758 3.39 3.5 8.792 12.897 18.184 18.825 30.354 19.16 12.067.332 14.908-1.168 29.013-15.317C181.473 158.612 195.32 148 198.94 148c.912 0 2.224-.563 2.912-1.25.898-.896 5.704-1.244 16.95-1.23 15.634.02 15.722.033 21.698 3.156 9.06 4.735 14.04 8.73 19.248 15.436 4.637 5.97 4.75 6.04 6.237 3.94 2.74-3.865 12.845-14.052 13.94-14.052.592 0 1.078-.338 1.08-.75.012-1.222 6.068-4.334 12.393-6.366 8.55-2.747 24.897-2.706 33.558.084 3.55 1.144 6.7 2.407 7 2.806.3.4 2.138 1.534 4.086 2.522 5.085 2.577 11.184 7.778 21.458 18.298 11.824 12.107 15.94 14.606 25.015 15.19 9.73.623 17.925-2.324 26.59-9.563 1.888-1.57 7.062-8.87 9.733-13.72 1.95-3.54 1.386 5.22-.938 14.56-1.925 7.74-6.9 19.74-9.38 22.63-.56.66-2.145 2.88-3.52 4.95-3.75 5.63-14.84 16.58-20.5 20.25-12.477 8.08-26.436 12.77-40.5 13.6-8.476.5-23.706-.84-27.5-2.43-1.1-.46-4.025-1.38-6.5-2.04-2.475-.66-6.876-2.29-9.78-3.61-2.903-1.32-5.483-2.4-5.732-2.4-1.02 0-13.642-8.13-19.31-12.44-3.345-2.54-7.576-6.37-9.402-8.5l-3.32-3.88-3.18 3.79C257.95 210.93 243.87 222 242.16 222c-.545 0-1.14.395-1.325.878-.183.483-3.933 2.68-8.333 4.882-19.038 9.526-39.892 13.008-58.5 9.767zm-1.594-159.52c-8.636-.944-14.453-3.438-20.44-8.763-11.826-10.52-14.99-29.243-7.322-43.33 3.49-6.416 8.05-10.408 16.31-14.276 5.78-2.71 7.6-3.07 15.546-3.075 7.427-.006 9.75.388 13.29 2.255 5.43 2.866 8.203 7.32 8.203 13.182 0 7.234-4.917 13.106-12.993 15.52-2.437.727-2.455.695-.72-1.25.98-1.097 2.212-3.5 2.74-5.34.846-2.948.645-3.67-1.696-6.113-8.034-8.386-20.356-4.395-23.882 7.736-2.124 7.304-1.174 11.553 3.535 15.807 4.824 4.358 9.81 5.97 16.88 5.453 8.677-.634 14.04-3.842 18.92-11.315 9.25-14.173 24.976-20.235 45.804-17.66 4.91.608 10.27 1.42 11.92 1.806 3.86.903 7.14.91 11.5.026 24.76-5.018 44.31-1.46 53.48 9.733 1.94 2.367 3.53 4.804 3.53 5.417 0 2.032 7.18 8.325 11.71 10.267 9 3.856 21.6 1.406 26.5-5.15 8.07-10.805-3.26-29.497-15.79-26.04-7.94 2.195-10.78 8.818-6.24 14.59l2.17 2.754-2.42-.72c-3.86-1.15-8.41-4.57-10.88-8.17-1.85-2.71-2.19-4.27-1.81-8.28.98-10.11 7.12-14.39 20.78-14.46 15.78-.08 27.03 6.4 33.66 19.41 2.46 4.83 2.8 6.61 2.8 14.5 0 11.84-3.12 19.13-11.48 26.87-7.8 7.22-14 9.01-30.97 8.92-25.37-.13-75.41-11.67-79.54-18.34-.44-.72-1.4-.42-2.81.88-6.42 5.89-50.5 16.3-74.15 17.51-4.95.26-12.19.11-16.09-.31z"/>
</svg>
</div>
The SVG is held within <a-assets></a-assets> along with other assets.
The entity's code looks like this:
<a-entity geometry="primitive: plane" position="10 4 -4" scale="4 4 0" draw="width: 512; height: 512;" htmltexture="asset: #bears">
</a-entity>
The rendered plane holding the SVG texture ends up having a white background. I tried changing the material transparency property, but that didn't work. I also tried changing the draw background property, seeing as how aframe-htmltexture-component has a dependency on aframe-draw-component which creates a canvas element to draw to, but again no luck.
Anyone know a fix for this, or have a better way at rendering HTML (SVG) as an aframe entity?
Try material.opacity:
<a-entity
geometry="primitive: plane"
position="10 4 -4"
scale="4 4 0"
draw="width: 512; height: 512;"
htmltexture="asset: #bears"
material="opacity: 0.0; transparent: true">
</a-entity>
Edit: Transparent must be set to true too.
Did you try transparent="true" ?
I had similar issues with importing PNGs with transparency, and then discovered setting transparent="true" fixed it.
<a-plane position="0 0.01 0" rotation="-90 0 0" height="7" width="7"
src ="#metatron-gates-platform"
transparent ="true">
</a-plane>
I ran into this as well. Try changing the order of the svg entities. Because it is a 3d environment it renders farthest from the camera first, so place those entities first in your code.
I have files with columns like this. This sample input below is partial.
Please check below for main file link. Each file will have only two rows.
Gene 0.4% 0.7% 1.1% 1.4% 1.8% 2.2% 2.5% 2.9% 3.3% 3.6% 4.0% 4.3% 4.7% 5.1% 5.4% 5.8% 6.2% 6.5% 6.9% 7.2% 7.6% 8.0% 8.3% 8.7% 9.1% 9.4% 9.8% 10.1% 10.5% 10.9% 11.2% 11.6% 12.0% 12.3% 12.7% 13.0% 13.4% 13.8% 14.1% 14.5% 14.9% 15.2% 15.6% 15.9% 16.3% 16.7% 17.0% 17.4% 17.8% 18.1% 18.5% 18.8% 19.2% 19.6% 19.9% 20.3% 20.7% 21.0% 21.4% 21.7% 22.1% 22.5% 22.8% 23.2% 23.6% 23.9% 24.3% 24.6% 25.0% 25.4% 25.7% 26.1% 26.4% 26.8% 27.2% 27.5% 27.9% 28.3% 28.6% 29.0% 29.3% 29.7% 30.1% 30.4% 30.8% 31.2% 31.5% 31.9% 32.2% 32.6% 33.0% 33.3% 33.7% 34.1% 34.4% 34.8% 35.1% 35.5% 35.9% 36.2% 36.6% 37.0% 37.3% 37.7% 38.0% 38.4% 38.8% 39.1% 39.5% 39.9% 40.2% 40.6% 40.9% 41.3% 41.7% 42.0% 42.4% 42.8% 43.1% 43.5% 43.8% 44.2% 44.6% 44.9% 45.3% 45.7% 46.0% 46.4% 46.7% 47.1% 47.5% 47.8% 48.2% 48.6% 48.9% 49.3% 49.6% 50.0% 50.4% 50.7% 51.1% 51.4% 51.8% 52.2% 52.5% 52.9% 53.3% 53.6% 54.0% 54.3% 54.7% 55.1% 55.4% 55.8% 56.2% 56.5% 56.9% 57.2% 57.6% 58.0% 58.3% 58.7% 59.1% 59.4% 59.8% 60.1% 60.5% 60.9% 61.2% 61.6% 62.0% 62.3% 62.7% 63.0% 63.4% 63.8% 64.1% 64.5% 64.9% 65.2% 65.6% 65.9% 66.3% 66.7% 67.0% 67.4% 67.8% 68.1% 68.5% 68.8% 69.2% 69.6% 69.9% 70.3% 70.7% 71.0% 71.4% 71.7% 72.1% 72.5% 72.8% 73.2% 73.6% 73.9% 74.3% 74.6% 75.0% 75.4% 75.7% 76.1% 76.4% 76.8% 77.2% 77.5% 77.9% 78.3% 78.6% 79.0% 79.3% 79.7% 80.1% 80.4% 80.8% 81.2% 81.5% 81.9% 82.2% 82.6% 83.0% 83.3% 83.7% 84.1% 84.4% 84.8% 85.1% 85.5% 85.9% 86.2% 86.6% 87.0% 87.3% 87.7% 88.0% 88.4% 88.8% 89.1% 89.5% 89.9% 90.2% 90.6% 90.9% 91.3% 91.7% 92.0% 92.4% 92.8% 93.1% 93.5% 93.8% 94.2% 94.6% 94.9% 95.3% 95.7% 96.0% 96.4% 96.7% 97.1% 97.5% 97.8% 98.2% 98.6% 98.9% 99.3% 99.6% 100.0% 0.4% 0.7% 1.1% 1.4% 1.8% 2.2% 2.5% 2.9% 3.3% 3.6% 4.0% 4.3% 4.7% 5.1% 5.4% 5.8% 6.2% 6.5% 6.9% 7.2% 7.6% 8.0% 8.3% 8.7% 9.1% 9.4% 9.8% 10.1% 10.5% 10.9% 11.2% 11.6% 12.0% 12.3% 12.7% 13.0% 13.4% 13.8% 14.1% 14.5% 14.9% 15.2% 15.6% 15.9% 16.3% 16.7% 17.0% 17.4% 17.8% 18.1% 18.5% 18.8% 19.2% 19.6% 19.9% 20.3% 20.7% 21.0% 21.4% 21.7% 22.1% 22.5% 22.8% 23.2% 23.6% 23.9% 24.3% 24.6% 25.0% 25.4% 25.7% 26.1% 26.4% 26.8% 27.2% 27.5% 27.9% 28.3% 28.6% 29.0% 29.3% 29.7% 30.1% 30.4% 30.8% 31.2% 31.5% 31.9% 32.2% 32.6% 33.0% 33.3% 33.7% 34.1% 34.4% 34.8% 35.1% 35.5% 35.9% 36.2% 36.6% 37.0% 37.3% 37.7% 38.0% 38.4% 38.8% 39.1% 39.5% 39.9% 40.2% 40.6% 40.9% 41.3% 41.7% 42.0% 42.4% 42.8% 43.1% 43.5% 43.8% 44.2% 44.6% 44.9% 45.3% 45.7% 46.0% 46.4% 46.7% 47.1% 47.5% 47.8% 48.2% 48.6% 48.9% 49.3% 49.6% 50.0% 50.4% 50.7% 51.1% 51.4% 51.8% 52.2% 52.5% 52.9% 53.3% 53.6% 54.0% 54.3% 54.7% 55.1% 55.4% 55.8% 56.2% 56.5% 56.9% 57.2% 57.6% 58.0% 58.3% 58.7% 59.1% 59.4% 59.8% 60.1% 60.5% 60.9% 61.2% 61.6% 62.0% 62.3% 62.7% 63.0% 63.4% 63.8% 64.1% 64.5% 64.9% 65.2% 65.6% 65.9% 66.3% 66.7% 67.0% 67.4% 67.8% 68.1% 68.5% 68.8% 69.2% 69.6% 69.9% 70.3% 70.7% 71.0% 71.4% 71.7% 72.1% 72.5% 72.8% 73.2% 73.6% 73.9% 74.3% 74.6% 75.0% 75.4% 75.7% 76.1% 76.4% 76.8% 77.2% 77.5% 77.9% 78.3% 78.6% 79.0% 79.3% 79.7% 80.1% 80.4% 80.8% 81.2% 81.5% 81.9% 82.2% 82.6% 83.0% 83.3% 83.7% 84.1% 84.4% 84.8% 85.1% 85.5% 85.9% 86.2% 86.6% 87.0% 87.3% 87.7% 88.0% 88.4% 88.8% 89.1% 89.5% 89.9% 90.2% 90.6% 90.9% 91.3% 91.7% 92.0% 92.4% 92.8% 93.1% 93.5% 93.8% 94.2% 94.6% 94.9% 95.3% 95.7% 96.0% 96.4% 96.7% 97.1% 97.5% 97.8% 98.2% 98.6% 98.9% 99.3% 99.6% 100.0%
Basically, here is what I need to be done.
a. Start from second column which is 0.4% here.
b. Go until you hit "10" in the header name. If the header name is exactly 10.0%, then include that column too. If not, only include until the column before it. In this example, since we have 10.1% (29th column), we will be including columns starting from 0.4%(second) until 9.8% which is the 28th column. If the 29th column was to be 10.0%, then it would have been included too.
c. Average the values for these respective columns in the second row (data is not presented here - please click this link for total dataset - https://goo.gl/W8jND7). In this example, starting from 0.4%(second column) till 9.8%(28th column).
d. In the output, print first column which is "Gene", and this average value with column header being
Gene Average_10%
e. Then start from 10.1% (29th column) and check until you hit "20" in the header name. Repeat steps b through d. And print output as
Gene Average_10% Average_20%
Repeat this until you have
Gene Average_10% Average_20% Average_30% Average_40% Average_50% Average_60% Average_70% Average_80% Average_90% Average_100%
f. After you hit 100%, it means one dataset is done.
g. If you observe my column header carefully here, there is another 0.4%-100% columns after the first 100%. I will be having 13 of these 0.4%-100%s in the input file at the above link.
i. I have multiple files, the headers can be
1% 2% 3%....100%
1.5% 2.5% 3.5%....100%
It varies from file to file. But the logic of averaging(if you hit "10", "20", etc) is always the same. And the number of samples 13 is also same which means each file will have 100%s for 13 times.
I should say, it's a horrible format for this task. I don't expect anyone to come up with a final solution for you but this is how I would approach this
awk 'NR == 1 {
gsub("%","");
for (f=2; f<=NF; f++) {
for (i=1; i<10; i++)
if ($f<10*i && $(f+1)>=10*i) print f, $f
if ($f==100) print f, $f
}}' file
28 9.8
56 19.9
83 29.7
111 39.9
138 49.6
166 59.8
194 69.9
221 79.7
249 89.9
277 100.0
304 9.8
332 19.9
359 29.7
387 39.9
414 49.6
442 59.8
470 69.9
497 79.7
525 89.9
553 100.0
here printing the column index and the threshold used for verification purposes. Once you have the column boundaries extracted it should be straightforward to sum the respective columns. Note that by your logic 100% should be never included, however it seems wrong so I have special case for it.
I have a list of data (two columns) and I want to plot in gnuplot only the value for which my second column has a local maximum.
To do so I would like to see if the second column of the ith row is bigger than the (i-1) and (i+1)th rows.
It can be done and I was bored enough to do it. I generated the following set of random data:
5191
29375
23222
32118
3185
32355
17173
8734
28850
20811
5956
6950
28560
25770
4630
28272
10035
7209
19428
26187
30784
20326
12865
23288
20924
Plotting the values against their position in the list looks like this:
You can spot the local maxima right away from the graph above. Now I can process the data points storing the two previous values (both x and y coordinates) in temporary variables, when I identify a maximum, I plot the data point:
# Select the columns of your data file that contain x and y data
# (usually 1 and 2 respectively)
xcolumn=0
ycolumn=1
plot "data" u (column(xcolumn)):(column(ycolumn)) w l, \
"data" u (column(0)==0 ? (last2y=column(ycolumn), \
last2x=column(xcolumn), 1/0) : column(0)==1 ? (lasty=column(ycolumn), \
lastx=column(xcolumn), 1/0) : lastx) \
: \
( column(0) < 2 ? 1/0 : (last2y < lasty && \
column(ycolumn) < lasty) ? (value=lasty, last2y=lasty, last2x=lastx, \
lasty=column(ycolumn), lastx=column(xcolumn), value) : (last2y=lasty, \
last2x=lastx, lasty=column(ycolumn), lastx=column(xcolumn), 1/0)) pt 7
A simple gnuplot peak finder
Comparing 3 consecutive data points is a simple way to look for maxima/peaks.
if y(i-1) < y(i) > y(i+1) then you have a maximum/peak at x(i).
However, if you have noisy curves (like experimental spectra typically are) you will get too many "peaks".
The script below basically calculates for each peak a specific number which is a measure for the "independence" of a peak (see Topographic prominence). Note that a high peak intensity does not necessarily mean high prominence. For the script gnuplot>=5.2 is required since it uses datablocks and plot ... with table. With some adaptions, it can maybe run with older versions.
You can give "threshold" to show only a certain number of peaks. Unfortunately, gnuplot does not have an easy internal feature for sorting datablocks, with this one could have easily shown the first N most prominent peaks. Hence, here it is done differently:
Threshold=100 means show all peaks, Threshold=0 means show only the global maximum. Note that the threshold values and number of shown peaks are not linear between 0 and 100 since it depends on the spectral shape.
Comments and improvements are welcome.
Data: SO28173128.dat
# x y
6.02 3.33
9.59 4.03
9.59 4.03
9.59 4.03
9.83 3.66
10.58 3.22
10.62 3.85
11.32 2.33
11.53 3.67
11.88 3.27
13.28 3.97
13.42 4.35
14.56 2.99
14.75 3.50
15.23 3.91
15.59 3.58
16.56 3.85
16.70 4.49
16.97 3.98
17.23 4.49
17.37 3.73
17.90 4.24
17.93 4.91
18.80 4.23
18.83 4.64
20.37 3.59
20.48 3.99
21.70 3.19
22.29 4.36
22.41 3.38
23.34 5.04
23.41 5.38
24.38 4.71
24.58 4.96
25.40 5.13
25.41 4.70
26.20 4.21
26.36 4.84
26.55 4.11
26.79 4.73
27.02 4.13
27.54 4.51
27.55 5.17
28.08 5.42
28.33 4.97
28.61 5.17
29.06 5.10
29.12 4.26
29.61 3.61
30.03 4.84
30.58 5.30
30.68 4.72
31.21 5.82
31.99 5.83
32.58 6.56
33.03 6.23
33.51 6.79
33.73 8.88
34.00 7.85
34.17 9.82
34.32 11.24
34.55 10.78
34.85 12.58
35.14 13.84
35.50 18.07
35.85 18.40
36.55 30.70
36.75 29.27
37.16 28.78
38.15 18.93
38.29 20.52
38.83 14.96
39.63 11.39
40.17 8.67
40.48 9.22
40.88 9.75
40.95 8.77
41.45 10.86
41.58 10.33
41.84 11.12
42.51 12.83
43.33 15.09
43.43 14.20
43.63 14.56
44.22 14.76
44.92 12.58
45.71 9.40
45.98 9.84
47.03 7.01
47.53 7.15
47.95 5.48
48.08 6.21
48.56 4.51
48.64 5.11
49.14 3.69
49.31 4.27
49.86 4.71
50.14 4.13
50.33 5.83
50.37 5.36
51.00 4.60
51.37 4.75
52.00 4.91
52.01 4.34
54.28 3.83
54.46 4.33
54.83 3.19
55.58 5.30
55.86 4.95
55.92 6.48
56.40 5.89
56.59 5.16
57.56 5.96
57.64 5.21
58.31 5.36
58.38 6.08
58.92 5.84
59.00 5.28
59.57 5.96
59.94 5.37
61.31 6.07
61.32 6.78
61.96 7.34
62.37 6.89
63.14 9.45
63.54 9.51
63.80 11.03
63.93 11.56
64.17 11.12
64.84 12.38
65.14 16.28
65.64 18.07
66.22 19.06
66.42 17.70
66.56 18.40
67.10 17.20
67.24 18.00
67.77 16.82
68.04 15.80
68.15 16.88
68.43 14.98
68.86 16.28
69.33 16.55
70.43 29.11
70.93 48.16
71.62 58.09
72.01 66.42
72.43 73.37
72.79 76.74
72.84 81.24
73.07 79.85
73.15 72.04
73.32 74.49
73.58 61.26
73.71 65.36
73.98 51.60
73.98 58.75
74.25 52.93
74.53 39.70
74.90 34.01
75.18 34.80
76.67 23.29
77.05 23.82
77.15 20.94
77.27 27.13
77.52 25.20
77.70 29.76
78.17 29.96
78.36 33.33
78.73 37.05
78.86 34.67
79.03 39.90
79.38 39.70
79.59 40.62
79.88 39.17
80.59 36.52
80.60 33.61
81.26 27.53
81.67 23.69
82.07 23.69
82.90 15.37
83.94 14.04
84.25 14.42
85.01 11.26
85.22 12.18
85.77 11.19
85.89 9.97
86.46 10.72
86.56 11.78
87.06 10.56
87.19 11.53
87.74 9.60
87.89 10.64
88.30 8.74
88.56 9.27
88.98 8.08
89.25 8.70
90.12 9.15
90.14 8.34
90.99 9.59
91.06 8.86
91.72 9.01
91.74 8.48
92.43 7.68
92.48 8.34
93.15 7.68
93.54 8.34
93.72 7.61
94.20 8.00
94.39 8.64
94.82 7.83
95.00 8.40
95.43 6.77
95.66 7.39
95.81 5.44
95.84 6.25
96.53 7.70
96.55 6.70
97.22 9.14
97.47 8.13
97.53 6.75
98.57 7.83
98.85 7.40
98.92 8.45
99.35 8.25
99.40 7.77
100.04 8.33
100.14 7.91
101.00 8.30
101.04 7.80
101.94 7.48
102.02 8.20
102.53 6.56
102.58 7.84
103.23 7.75
103.38 6.82
103.39 8.93
103.75 8.60
104.05 9.26
104.30 8.14
104.55 8.56
104.85 7.15
105.17 10.09
105.21 8.41
105.52 11.78
105.80 10.00
105.92 10.86
106.11 9.54
106.31 9.93
106.86 10.99
107.20 10.88
107.42 12.05
107.62 10.33
107.80 11.16
108.39 9.90
108.79 12.25
109.08 11.85
109.46 13.77
109.73 13.42
109.96 15.16
111.02 20.08
111.30 23.56
111.69 23.56
112.35 30.08
112.61 39.04
112.89 43.53
113.28 43.67
113.81 60.73
113.94 56.63
114.20 63.11
114.46 69.60
115.00 80.44
115.12 86.95
115.27 84.68
115.78 88.78
116.11 89.57
116.40 88.51
116.43 89.90
117.90 80.31
118.18 71.32
118.42 72.51
118.70 67.48
118.83 61.13
119.25 60.34
119.37 54.51
119.65 55.18
120.08 50.28
120.12 48.96
120.84 54.18
120.95 51.87
121.24 56.49
121.50 53.01
121.64 54.44
121.73 51.74
121.84 52.53
122.14 53.72
122.19 52.40
122.44 53.59
122.45 54.91
122.64 51.61
122.76 53.17
123.08 50.79
123.10 50.15
123.34 49.51
123.39 50.81
123.88 50.28
124.25 46.58
124.26 44.06
124.53 43.67
125.18 38.77
125.81 32.16
126.10 32.55
127.32 21.71
127.70 21.18
127.96 18.27
128.22 18.66
128.62 15.71
128.89 16.24
129.30 13.94
129.56 14.47
129.96 12.22
130.21 12.72
130.53 10.83
130.93 10.61
131.15 11.24
131.84 10.06
131.93 11.12
132.71 9.87
133.06 13.17
133.38 12.73
134.17 16.97
134.31 22.37
134.85 30.83
135.38 43.01
135.62 40.09
135.64 50.81
135.75 54.05
136.16 56.10
136.80 50.22
137.21 49.36
138.29 25.41
138.56 26.73
139.98 13.09
140.03 11.92
140.32 14.53
140.55 13.80
140.85 16.25
141.24 15.81
141.45 17.92
141.59 16.51
141.99 20.65
142.25 20.19
142.76 27.79
142.91 24.95
143.72 37.66
143.79 35.37
144.65 29.11
144.91 30.32
145.71 25.74
145.84 24.80
146.54 27.73
146.74 30.57
147.30 31.11
147.42 30.30
148.33 31.23
148.38 30.68
148.93 31.19
149.09 31.74
149.24 28.43
149.49 29.03
149.96 24.55
150.69 23.48
151.83 18.06
151.89 19.39
152.41 18.27
152.96 19.45
153.02 18.53
153.47 19.42
153.71 18.78
153.91 19.44
154.08 17.74
154.54 18.92
154.65 18.20
155.61 18.79
156.78 15.75
157.06 16.08
157.54 12.68
157.84 13.40
158.51 12.73
158.57 11.85
159.45 11.27
159.57 10.33
160.11 8.81
160.24 9.81
160.90 9.27
160.90 8.08
161.17 8.48
161.43 8.10
161.59 8.72
161.83 7.91
162.44 8.48
162.61 7.94
163.41 8.75
163.43 8.19
163.94 8.99
163.96 8.33
164.60 8.23
164.87 8.74
165.81 8.50
166.10 8.08
166.74 8.91
167.02 8.48
167.67 9.68
167.96 9.28
168.90 9.81
168.96 10.34
169.57 9.77
169.76 11.28
170.00 10.75
171.01 13.77
171.12 13.13
171.79 13.71
172.83 19.14
173.19 19.13
173.52 21.06
173.73 20.38
173.87 21.65
174.27 21.90
174.41 25.94
174.94 27.39
175.20 33.48
175.71 35.07
175.84 40.49
176.48 43.27
176.51 46.84
177.34 51.47
177.62 57.30
177.69 55.18
177.87 53.26
178.08 54.46
178.45 49.43
178.76 49.62
179.02 43.53
179.43 42.08
179.96 28.98
180.21 26.40
180.56 26.57
180.57 26.20
181.14 26.43
181.24 25.51
181.68 24.93
181.74 27.13
181.97 28.85
182.05 26.93
182.14 29.78
182.30 27.53
182.59 28.72
182.72 26.93
183.04 27.67
183.21 28.76
183.33 27.92
183.75 29.99
184.06 29.78
184.28 32.07
184.71 32.49
185.40 37.78
185.61 34.01
186.02 33.86
186.20 30.07
186.79 28.72
186.97 25.44
187.34 25.15
188.45 17.65
188.84 16.25
189.08 16.77
189.50 14.08
189.90 13.69
189.92 11.67
190.37 12.04
190.41 12.90
190.90 11.40
190.93 12.58
192.29 9.27
192.33 8.79
193.23 10.21
193.34 9.73
193.78 11.15
194.07 10.20
194.54 10.43
194.93 9.64
195.39 9.82
195.98 10.86
196.08 10.05
196.45 11.78
197.00 8.90
197.03 9.95
197.70 7.15
198.07 10.13
198.10 8.48
198.83 9.67
198.85 10.46
199.27 9.80
199.46 9.46
200.46 11.78
200.73 11.38
201.38 12.37
201.92 9.72
202.06 10.52
203.14 7.67
203.52 7.95
203.93 7.02
204.13 7.55
204.39 6.66
204.63 7.15
206.35 5.53
206.57 6.00
207.21 6.57
207.33 5.76
208.23 7.93
208.52 7.68
208.92 7.15
209.21 7.56
209.41 6.62
209.88 6.89
211.48 4.50
211.70 5.00
212.39 5.37
212.51 4.53
212.84 4.31
212.92 5.17
213.26 4.38
213.44 5.24
213.70 4.57
214.15 4.97
214.24 5.71
215.55 5.20
215.65 6.01
216.28 6.16
216.55 6.87
216.73 5.56
216.95 6.23
217.25 6.03
217.47 5.46
217.93 5.50
218.26 5.83
218.56 5.04
219.31 4.73
219.97 4.72
220.10 5.61
220.79 4.23
220.87 4.94
221.87 4.64
222.07 4.12
222.58 4.47
222.69 5.02
223.21 4.51
223.24 5.22
223.86 4.54
223.90 5.22
224.48 4.29
224.54 5.04
225.22 4.73
225.28 3.86
225.54 4.13
225.76 4.72
225.84 3.93
226.52 4.23
226.60 4.94
226.98 4.02
227.78 3.89
228.29 4.67
228.62 4.29
229.12 4.90
229.46 4.16
229.83 4.48
230.28 3.54
230.50 4.06
230.91 3.03
231.18 3.63
231.79 3.99
231.97 3.45
232.70 4.86
Edit: revised and simplified
Script: (works with gnuplot>=5.2.0)
### a simple gnuplot peak finder
reset session
FILE = "SO28173128.dat"
colX = 1
colY = 2
# extract all peaks
set table $Temp
plot y2=y1=x2=x1=NaN FILE u (x0=x1,x1=x2,x2=column(colX), \
y0=y1,y1=y2,y2=column(colY), \
y0<y1 && y1>=y2 ? sprintf("%g %g",x1,y1) : '') w table
set table $Peaks # remove empty lines and store first values in x0,y0 for later use
plot $Temp u (column(-2)==0 && $0==0?x0=$1:$1):(column(-2)==0 && $0==0?y0=$2:$2) w table
unset table
isNaN(v) = v!=v # check if value is NaN
min(a,b) = isNaN(a) && isNaN(b) ? NaN : isNaN(a) ? b : isNaN(b) ? a : a<b ? a : b # get minimum incl. NaN
# create prominence table
set print $Prominence
do for [n0=0:|$Peaks|-1] {
xn = yn = xp = yp = NaN
stats $Peaks u ($0<n0 && $2>y0 ? (xn=$1,yn=$2) : 0, \
$0>n0 && $2>y0 && yp!=yp ? (xp=$1,yp=$2) : 0, \
$0==n0+1 ? (x1=$1, y1=$2) : 0 ) nooutput
print sprintf("%g %g %g",x0,y0, min(x0-xn,xp-x0))
x0=x1
y0=y1
}
set print
set key noautotitle
set offsets 0,0,1,0
set grid y
# filter Peaks
stats $Prominence u 3 name "P" nooutput # get min, max
Filter(col,t) = (!valid(col) ? 0 : (1-(column(col)-P_min)/(P_max-P_min))*99+1)<=t ? $2 : NaN
Threshold = 100
set label 1 at graph 0.02,0.95 sprintf("Threshold: %g",Threshold)
plot FILE u colX:colY w l lc rgb "blue" ti "Spectrum", \
$Prominence u 1:(Filter(3,Threshold)) w impulses lc rgb "red", \
$Prominence u 1:(Filter(3,Threshold)):1 w labels offset 0,1
### end of script
Results: (for different thresholds)
Let us consider following stock prices taken from yahoo.finance.com:
Date Open High Low Close Volume Adj Close
3/4/2013 23.15 23.84 23.03 23.67 30908300 23.3
2/25/2013 23.5 23.53 22.81 23.19 40710800 22.83
2/19/2013 23.42 23.75 23.12 23.39 38743400 23.03
2/11/2013 22.49 23.55 22.35 23.29 46448500 22.74
2/4/2013 22.41 22.62 22.27 22.5 34498100 21.97
1/28/2013 22.44 22.64 22.18 22.62 39634900 22.09
1/22/2013 22.18 22.31 21.75 22.29 47826300 21.77
1/14/2013 21.18 22.19 21.01 22.04 54826000 21.52
1/7/2013 21.16 21.24 20.68 21.13 35304100 20.63
12/31/2012 20.29 21.54 20.26 21.2 45796500 20.7
12/24/2012 20.79 20.96 20.42 20.44 28597100 19.96
12/17/2012 21.69 21.95 20.56 20.88 70719700 20.39
12/10/2012 21.43 21.95 21.36 21.62 39455500 20.92
12/3/2012 21.18 21.48 20.71 21.46 35913000 20.77
11/26/2012 20.88 21.36 20.5 21.13 36203100 20.45
11/19/2012 20.41 21.04 20.37 21.04 35401500 20.36
11/12/2012 21.04 21.14 19.87 20.15 45095400 19.5
11/5/2012 21.2 21.78 20.7 21 37812800 20.32
11/2/2012 21.53 21.68 21.26 21.31 47475200 20.62
And i want to do correlation matrix between for example Volume and Low Close variable. I used correlation function from data analysis toolbox from excel, but I got only one side matrix, like this:
Close Volume
Close 1
Volume -0.117267345 1
It does not show me correlation coefficients up side of main diagonal, why? Maybe it is symmetric and because of this?
The correlation matrix is necessarily symmetric, so the above the diagonal element in your case is -0.117257345. If you check the documentation on the correl function, and look at the defining equation, you can see that it is symmetric with respect to exchanging X<->Y
v0.10.4
Here's the simple loop that results in an ever-increasing memory usage:
function redx(){
setTimeout(function(){ redx() },1000);
console.log('loop');
}
redx();
What am I doing wrong ??
EDIT
OK, just tried the suggestion to reference the timeout object in the scope and it seems that garbage collection does kick in after about 40 seconds, here's abbreviated logs from TOP:
3941 root 20 0 32944 7284 4084 S 4.587 3.406 0:01.32 node
3941 root 20 0 32944 7460 4084 S 2.948 3.489 0:01.59 node
3941 root 20 0 32944 7516 4084 S 2.948 3.515 0:01.68 node
3941 root 20 0 33968 8400 4112 S 2.948 3.928 0:02.15 node
3941 root 20 0 33968 8920 4112 S 3.275 4.171 0:02.98 node
3941 root 20 0 33968 8964 4112 S 2.948 4.192 0:03.07 node
3941 root 20 0 33968 9212 4112 S 2.953 4.308 0:03.16 node
3941 root 20 0 33968 9212 4112 S 2.953 4.308 0:03.25 node
3941 root 20 0 33968 9212 4112 S 3.276 4.308 0:03.35 node
3941 root 20 0 33968 9212 4112 S 2.950 4.308 0:03.44 node
No idea why but apparently if you reference the timeout object in the scope of the function nodejs will do the garbage collect that correctly.
function redx(){
var t = setTimeout(function(){ redx() },50);
console.log('hi');
}
redx();
Actually, I think it might be just the way the V8 garbage collector works.
On my system, node heap tends to increase up to 48 MB and then stabilize, so I think if you keep your program running for a long time, the memory consumption will eventually stabilize.
You can have information about when/how the GC kicks in by launching node with one of the V8 command line option: the --trace_gc flag.
In your first tries with Redis, you were systematically connecting/disconnecting from Redis at each call. This tends to generate garbage. You are supposed to open a connection once and use it many times. Nevertheless, even when I do this, memory consumption tends to stabilize. Here is the evolution of memory consumption on this example with Redis:
// something close to your initial function (when Redis was still in the picture)
function redx(){
var client = redis.createClient();
client.get("tally", function(err, reply) {
client.quit();
});
setTimeout(function(){ redx() }, 50 );
}
Here, the stabilization after 60 MB seems to be quite obvious.