I am using Rollup with Vite and vite-plugin-ssr.
The production build it produces with standard configuration includes a ton of tiny files, e.g.
dist/client/assets/chunk-adbd3755.js 0.69 KiB / gzip: 0.43 KiB
dist/client/assets/chunk-adbd3755.js.map 0.10 KiB
dist/client/assets/chunk-d703547b.js 2.35 KiB / gzip: 1.15 KiB
dist/client/assets/chunk-d703547b.js.map 7.79 KiB
dist/client/assets/chunk-860778e7.js 0.95 KiB / gzip: 0.51 KiB
dist/client/assets/chunk-860778e7.js.map 3.26 KiB
dist/client/assets/chunk-90693398.js 2.37 KiB / gzip: 0.94 KiB
dist/client/assets/chunk-90693398.js.map 10.18 KiB
dist/client/assets/chunk-a3b7c495.js 2.21 KiB / gzip: 1.04 KiB
dist/client/assets/chunk-a3b7c495.js.map 5.33 KiB
dist/client/assets/chunk-d321b675.js 0.67 KiB / gzip: 0.46 KiB
dist/client/assets/chunk-d321b675.js.map 4.34 KiB
dist/client/assets/chunk-c5fff223.js 0.36 KiB / gzip: 0.23 KiB
dist/client/assets/chunk-c5fff223.js.map 1.17 KiB
dist/client/assets/chunk-93f6d1e0.js 2.42 KiB / gzip: 0.93 KiB
dist/client/assets/chunk-93f6d1e0.js.map 6.81 KiB
dist/client/assets/chunk-84638231.js 0.90 KiB / gzip: 0.52 KiB
dist/client/assets/chunk-84638231.js.map 2.43 KiB
dist/client/assets/chunk-7b2a7fc5.js 16.33 KiB / gzip: 3.33 KiB
I am assuming this is determined by Rollup configuration, and I would like to force rollup to target producing chunks that are at least 5KiB in size, i.e. if there are multiple small chunks, they should be put in the same file.
How do I do that?
At the same time, there are a few very large chunks that would benefit from code splitting:
dist/client/assets/chunk-79a47c16.js 233.29 KiB / gzip: 65.45 KiB
dist/client/assets/chunk-79a47c16.js.map 1090.09 KiB
dist/client/assets/chunk-338b0152.js 295.43 KiB / gzip: 64.69 KiB
dist/client/assets/chunk-338b0152.js.map 726.44 KiB
dist/client/assets/chunk-4bad2a7c.js 850.46 KiB / gzip: 175.06 KiB
dist/client/assets/chunk-4bad2a7c.js.map 1842.03 KiB
dist/client/assets/chunk-00222315.js 1220.59 KiB / gzip: 341.05 KiB
dist/client/assets/chunk-00222315.js.map 3558.01 KiB
dist/client/assets/chunk-41093922.js 1387.07 KiB / gzip: 364.90 KiB
dist/client/assets/chunk-41093922.js.map 5215.41 KiB
Same question but inverse: How do I split them into smaller files?
Related
I need to complete this exercise:
https://codelabs.developers.google.com/codelabs/mdc-102-web#2
But after all of these steps I get an error:
PS C:\Users\yuriy\Documents\GitHub\material\mdc-101\starter> npm start
> mdc-101-web#0.0.0 start C:\Users\yuriy\Documents\GitHub\material\mdc-101\starter
> webpack-dev-server --progress
10% [3] building modules 1/1 modules 0 active
Project is running at http://localhost:8081/
webpack output is served from /
[0] Hash: 555d1685cfed3b864a6318f77429ebbf5a38e07aeeac2c79f02ede37da1c2e8bbb7df60d48519ced
Version: webpack 3.12.0
Child
Hash: 555d1685cfed3b864a63
Time: 5520ms
Asset Size Chunks Chunk Names
bundle-login.css 56.6 kB [emitted]
style-bundle-login.js 335 kB 0 [emitted] [big] main
[2] multi (webpack)-dev-server/client?http://localhost:8081 ./login.scss 40 bytes {0} [built]
[3] (webpack)-dev-server/client?http://localhost:8081 7.93 kB {0} [built]
[4] ./node_modules/url/url.js 23.3 kB {0} [built]
[5] ./node_modules/node-libs-browser/node_modules/punycode/punycode.js 14.7 kB {0} [built]
[7] ./node_modules/url/util.js 314 bytes {0} [built]
[8] ./node_modules/querystring-es3/index.js 127 bytes {0} [built]
[11] ./node_modules/strip-ansi/index.js 161 bytes {0} [built]
[12] ./node_modules/ansi-regex/index.js 135 bytes {0} [built]
[13] ./node_modules/loglevel/lib/loglevel.js 8.86 kB {0} [built]
[14] (webpack)-dev-server/client/socket.js 1.08 kB {0} [built]
[15] ./node_modules/sockjs-client/dist/sockjs.js 181 kB {0} [built]
[16] (webpack)-dev-server/client/overlay.js 3.67 kB {0} [built]
[22] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {0} [built]
[24] (webpack)/hot/emitter.js 77 bytes {0} [built]
[26] ./login.scss 62 bytes {0} [built]
+ 12 hidden modules
Child
Hash: 18f77429ebbf5a38e07a
Time: 3083ms
Asset Size Chunks Chunk Names
style-bundle-home.js 335 kB 0 [emitted] [big] main
[2] multi (webpack)-dev-server/client?http://localhost:8081 ./home.scss 40 bytes {0} [built]
[3] (webpack)-dev-server/client?http://localhost:8081 7.93 kB {0} [built]
[4] ./node_modules/url/url.js 23.3 kB {0} [built]
[5] ./node_modules/node-libs-browser/node_modules/punycode/punycode.js 14.7 kB {0} [built]
[7] ./node_modules/url/util.js 314 bytes {0} [built]
[8] ./node_modules/querystring-es3/index.js 127 bytes {0} [built]
[11] ./node_modules/strip-ansi/index.js 161 bytes {0} [built]
[12] ./node_modules/ansi-regex/index.js 135 bytes {0} [built]
[13] ./node_modules/loglevel/lib/loglevel.js 8.86 kB {0} [built]
[14] (webpack)-dev-server/client/socket.js 1.08 kB {0} [built]
[15] ./node_modules/sockjs-client/dist/sockjs.js 181 kB {0} [built]
[16] (webpack)-dev-server/client/overlay.js 3.67 kB {0} [built]
[22] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {0} [built]
[24] (webpack)/hot/emitter.js 77 bytes {0} [built]
[26] ./home.scss 565 bytes {0} [built] [failed] [1 error]
+ 12 hidden modules
ERROR in ./home.scss
Module build failed:
undefined
^
Can't find stylesheet to import.
╷
22 │ #use '#material/theme/theme-color';
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
╵
node_modules\#material\list\_variables.scss 22:1 #forward
node_modules\#material\drawer\mdc-drawer.import.scss 13:1 #import
stdin 2:9 root stylesheet
in C:\Users\yuriy\Documents\GitHub\material\mdc-101\starter\node_modules\#material\list\_variables.scss (line 22, column 1)
# multi (webpack)-dev-server/client?http://localhost:8081 ./home.scss
Child
Hash: eeac2c79f02ede37da1c
Time: 2339ms
Asset Size Chunks Chunk Names
bundle-login.js 526 kB 0 [emitted] [big] main
[5] ./node_modules/#material/ripple/foundation.js 23.4 kB {0} [built]
[6] ./node_modules/#material/ripple/util.js 1.91 kB {0} [built]
[8] ./node_modules/#material/textfield/constants.js 2.81 kB {0} [built]
[24] multi (webpack)-dev-server/client?http://localhost:8081 ./login.js 40 bytes {0} [built]
[25] (webpack)-dev-server/client?http://localhost:8081 7.93 kB {0} [built]
[26] ./node_modules/url/url.js 23.3 kB {0} [built]
[33] ./node_modules/strip-ansi/index.js 161 bytes {0} [built]
[35] ./node_modules/loglevel/lib/loglevel.js 8.86 kB {0} [built]
[36] (webpack)-dev-server/client/socket.js 1.08 kB {0} [built]
[38] (webpack)-dev-server/client/overlay.js 3.67 kB {0} [built]
[44] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {0} [built]
[46] (webpack)/hot/emitter.js 77 bytes {0} [built]
[48] ./login.js 385 bytes {0} [built]
[49] ./node_modules/#material/textfield/index.js 1.36 kB {0} [built]
[61] ./node_modules/#material/ripple/index.js 1.29 kB {0} [built]
+ 47 hidden modules
Child
Hash: 2e8bbb7df60d48519ced
Time: 2142ms
Asset Size Chunks Chunk Names
bundle-home.js 437 kB 0 [emitted] [big] main
[0] ./node_modules/#material/list/constants.js 3.35 kB {0} [built]
[5] ./node_modules/#material/list/foundation.js 28.5 kB {0} [built]
[8] multi (webpack)-dev-server/client?http://localhost:8081 ./home.js 40 bytes {0} [built]
[9] (webpack)-dev-server/client?http://localhost:8081 7.93 kB {0} [built]
[10] ./node_modules/url/url.js 23.3 kB {0} [built]
[11] ./node_modules/node-libs-browser/node_modules/punycode/punycode.js 14.7 kB {0} [built]
[13] ./node_modules/url/util.js 314 bytes {0} [built]
[17] ./node_modules/strip-ansi/index.js 161 bytes {0} [built]
[19] ./node_modules/loglevel/lib/loglevel.js 8.86 kB {0} [built]
[20] (webpack)-dev-server/client/socket.js 1.08 kB {0} [built]
[22] (webpack)-dev-server/client/overlay.js 3.67 kB {0} [built]
[28] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {0} [built]
[30] (webpack)/hot/emitter.js 77 bytes {0} [built]
[32] ./home.js 110 bytes {0} [built]
[33] ./node_modules/#material/list/index.js 1.25 kB {0} [built]
+ 23 hidden modules
webpack: Failed to compile.
I have tried to use npm audit, but it doesn't work. I have tried to change fibers version to 5.0.0, but anyway it doesn't work. What should I change?
My code is available here:
https://github.com/yytrofimov/material
Both component should be installed in version 6.0.0:
npm install #material/drawer#^6.0.0
npm install #material/list#^6.0.0
I wrote a script which is fetching the HTML content of a page and analyze it.
Running this code in a loop, where I gave several URLs, I notice the memory usage was growing too much and too quickly.
Profiling and debugging the code with several tools, I notice the problem comes from the bit of code that is using BeautifulSoup4, at least I think.
Line Mem usage Increment Line Contents
59 40.5 MiB 40.5 MiB #profile
60 def crawl(self):
70 40.6 MiB 0.0 MiB self.add_url_to_crawl(self.base_url)
71
72 291.8 MiB 0.0 MiB for url in self.page_queue:
74 267.4 MiB 0.0 MiB if url in self.crawled_urls:
75 continue
76
77 267.4 MiB 0.0 MiB page = Page(url=url, base_domain=self.base_url, log=self.log)
78
79 267.4 MiB 0.0 MiB if page.parsed_url.netloc != page.base_domain.netloc:
80 continue
81
82 291.8 MiB 40.1 MiB page.analyze()
83
84 291.8 MiB 0.0 MiB self.content_hashes[page.content_hash].add(page.url)
94
95 # Add crawled page links into the queue
96 291.8 MiB 0.0 MiB for url in page.links:
97 291.8 MiB 0.0 MiB self.add_url_to_crawl(url)
98
100 291.8 MiB 0.0 MiB self.crawled_pages.append(page.getData())
101 291.8 MiB 0.0 MiB self.crawled_urls.add(page.url)
102
103 291.8 MiB 0.0 MiB mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
104 291.8 MiB 0.0 MiB print('Memory usage is: {0} KB'.format(mem))
Here is what line 104 is printing each run:
Memory usage is: 69216 KB
Memory usage is: 92092 KB
Memory usage is: 105796 KB
Memory usage is: 134704 KB
Memory usage is: 158604 KB
Memory usage is: 184068 KB
Memory usage is: 225324 KB
Memory usage is: 248708 KB
Memory usage is: 273780 KB
Memory usage is: 298768 KB
Using tracemalloc in the main file which is calling all the modules and running the crawl method above, I got the following list from tracemalloc.snapshot():
/usr/lib/python3.8/site-packages/bs4/element.py:744: size=23.3 MiB, count=210391, average=116 B
/usr/lib/python3.8/site-packages/bs4/builder/__init__.py:215: size=17.3 MiB, count=335036, average=54 B
/usr/lib/python3.8/site-packages/bs4/element.py:628: size=9028 KiB, count=132476, average=70 B
/usr/lib/python3.8/html/parser.py:327: size=7804 KiB, count=147140, average=54 B
/usr/lib/python3.8/site-packages/bs4/element.py:121: size=6727 KiB, count=132476, average=52 B
/usr/lib/python3.8/site-packages/bs4/element.py:117: size=6702 KiB, count=40848, average=168 B
/usr/lib/python3.8/html/parser.py:324: size=6285 KiB, count=85986, average=75 B
/usr/lib/python3.8/site-packages/bs4/element.py:772: size=5754 KiB, count=105215, average=56 B
/usr/lib/python3.8/html/parser.py:314: size=5334 KiB, count=105196, average=52 B
/usr/lib/python3.8/site-packages/bs4/__init__.py:587: size=4932 KiB, count=105197, average=48 B
Most of the files listed above are under the /bs4/ folder. Now, from the moment the page variable (line 82) is not stored anywhere, page.getData() is returning a dictionary and page.url a string, why do I have BeautifulSoup getting so much space in memory?
In line 72 you can see how the memory usage changed from ~40MB to ~291MB (considering the loop processed 10 URLs), it's a big change considering that the data I'm actually saving are a small dictionary and a string.
Am I having a problem with the garbage collector or I wrote something wrong?
I'm not very practical with Python, so I hope the point I've made with the profiling and debugging is correct.
i'm trying to optimize my react web app by using gzip compression. I was able to add the webpack plugin and do the compression. But my browser is not using the compressed files it seems. Please help me since I'm new to react. Also note that I use neutrinojs. I'll mention my bundle stats below.
Asset Size Chunks Chunk Names
assets/0.3fce4eb2.js 34.2 KiB 0 [emitted] [immutable]
assets/0.3fce4eb2.js.br 8.64 KiB [emitted]
assets/0.3fce4eb2.js.gz 9.6 KiB [emitted]
assets/0.3fce4eb2.js.map 133 KiB 0 [emitted] [dev]
assets/1.289d6022.js 33.9 KiB 1 [emitted] [immutable]
assets/1.289d6022.js.br 8.46 KiB [emitted]
assets/1.289d6022.js.gz 9.53 KiB [emitted]
assets/1.289d6022.js.map 98.1 KiB 1 [emitted] [dev]
assets/1.2db4754f.css 99 bytes 1 [emitted] [immutable]
assets/1.2db4754f.css.map 238 bytes 1 [emitted] [dev]
assets/10.b8f7f8a6.js 14.6 KiB 10 [emitted] [immutable]
assets/10.b8f7f8a6.js.br 5.12 KiB [emitted]
assets/10.b8f7f8a6.js.gz 5.68 KiB [emitted]
assets/10.b8f7f8a6.js.map 52.5 KiB 10 [emitted] [dev]
assets/11.14cee74f.js 856 bytes 11 [emitted] [immutable]
assets/11.14cee74f.js.map 3.07 KiB 11 [emitted] [dev]
assets/11.74159567.css 7.27 KiB 11 [emitted] [immutable]
assets/11.74159567.css.br 1.42 KiB [emitted]
assets/11.74159567.css.gz 1.66 KiB [emitted]
assets/11.74159567.css.map 8.37 KiB 11 [emitted] [dev]
assets/12.6d40dbcc.js 1.33 KiB 12 [emitted] [immutable]
assets/12.6d40dbcc.js.br 671 bytes [emitted]
assets/12.6d40dbcc.js.gz 787 bytes [emitted]
assets/12.6d40dbcc.js.map 1.96 KiB 12 [emitted] [dev]
assets/13.3d0e5b84.js 22.4 KiB 13 [emitted] [immutable]
assets/13.3d0e5b84.js.br 5.71 KiB [emitted]
assets/13.3d0e5b84.js.gz 6.55 KiB [emitted]
assets/13.3d0e5b84.js.map 87.2 KiB 13 [emitted] [dev]
assets/14.52749a43.js 8.25 KiB 14 [emitted] [immutable]
assets/14.52749a43.js.br 2.44 KiB [emitted]
assets/14.52749a43.js.gz 2.84 KiB [emitted]
assets/14.52749a43.js.map 19.6 KiB 14 [emitted] [dev]
assets/15.05afd908.js 21.1 KiB 15 [emitted] [immutable]
assets/15.05afd908.js.br 7.48 KiB [emitted]
assets/15.05afd908.js.gz 9.68 KiB [emitted]
assets/15.05afd908.js.map 154 KiB 15 [emitted] [dev]
assets/2.25cbca89.js 252 bytes 2 [emitted] [immutable]
assets/2.25cbca89.js.map 547 bytes 2 [emitted] [dev]
assets/3.36420eb7.css 618 bytes 3 [emitted] [immutable]
assets/3.36420eb7.css.map 938 bytes 3 [emitted] [dev]
assets/3.6d75eeb4.js 22.2 KiB 3 [emitted] [immutable]
assets/3.6d75eeb4.js.br 6.07 KiB [emitted]
assets/3.6d75eeb4.js.gz 6.91 KiB [emitted]
assets/3.6d75eeb4.js.map 62.9 KiB 3 [emitted] [dev]
assets/4.aa3c9f8e.js 81.1 KiB 4 [emitted] [immutable]
assets/4.aa3c9f8e.js.br 16.2 KiB [emitted]
assets/4.aa3c9f8e.js.gz 18.5 KiB [emitted]
assets/4.aa3c9f8e.js.map 265 KiB 4 [emitted] [dev]
assets/4.de23ee90.css 90 bytes 4 [emitted] [immutable]
assets/4.de23ee90.css.map 259 bytes 4 [emitted] [dev]
assets/7.0b2209a3.js 4.22 MiB 7 [emitted] [immutable] [big]
assets/7.0b2209a3.js.br 940 KiB [emitted] [big]
assets/7.0b2209a3.js.gz 1.19 MiB [emitted] [big]
assets/7.0b2209a3.js.map 14.6 MiB 7 [emitted] [dev]
assets/7.59caa759.css 259 KiB 7 [emitted] [immutable] [big]
assets/7.59caa759.css.br 23.6 KiB [emitted]
assets/7.59caa759.css.gz 30.6 KiB [emitted]
assets/7.59caa759.css.map 314 KiB 7 [emitted] [dev]
I really don't know why there are this much of files because I have seen only 2 to 4 files in examples. But I can see that there are gzip and brotli compressed files. But the browser says there are no content-encoding. I only added the plugin to config file and I think there is something more to do. Also I'm running this locally. Please help me with what could have gone wrong. Thank you.
If you have express js backend you can do something like below with compression package.in your main js file.
const compression = require('compression')
app.use(compression())
compression package
The floating point numbers with finite precision are represented with different precision in identical conditions
It is detected and tested on python version 3.x under Linux and Windows. And take the negative effect for the next calculation.
for i in range(100):
k = 1 + i / 100;
print(k)
1.0
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
1.1
1.11
1.12
1.13
1.1400000000000001
1.15
1.16
1.17
1.18
1.19
1.2
1.21
1.22
1.23
1.24
1.25
1.26
1.27
1.28
1.29
1.3
1.31
1.32
1.33
1.34
1.35
1.3599999999999999
1.37
1.38
1.3900000000000001
1.4
1.41
1.42
1.43
1.44
1.45
1.46
1.47
1.48
1.49
1.5
1.51
1.52
1.53
1.54
1.55
1.56
1.5699999999999998
1.58
1.5899999999999999
1.6
1.6099999999999999
1.62
1.63
1.6400000000000001
1.65
1.6600000000000001
1.67
1.6800000000000002
1.69
1.7
1.71
1.72
1.73
1.74
1.75
1.76
1.77
1.78
1.79
1.8
1.81
1.8199999999999998
1.83
1.8399999999999999
1.85
1.8599999999999999
1.87
1.88
1.8900000000000001
1.9
1.9100000000000001
1.92
1.9300000000000002
1.94
1.95
1.96
1.97
1.98
1.99
It is possible to set the precision in the following way:
for i in range(100):
k = 1 + i / 100;
print("%.Nf"%k)
Where N - decimal numbers.
Keep in mind, that regularly you don't need a lot of them, though the number could be really huge.
On CentOS 7.3.1611(linux kernel 3.10), My program is I/O bound that read disk device using buffer I/O, generally linux kernel may use the full memory to cache disk data as buffers, but it always left about 20G memory that not used, and kswapd daemon keeping to reclaim pages all the time.
top
top - 14:11:47 up 16 days, 2:42, 5 users, load average: 2.92, 3.18, 3.37
Tasks: 329 total, 2 running, 327 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.3 us, 7.7 sy, 0.0 ni, 83.3 id, 4.7 wa, 0.0 hi, 1.1 si, 0.0 st
KiB Mem : 13175558+total, 22444704 free, 26496388 used, 82814488 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 10222934+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
181 root 20 0 0 0 0 S 13.0 0.0 1395:34 kswapd0
cat /proc/meminfo
MemTotal: 131755580 kB
MemFree: 21335956 kB
MemAvailable: 102232248 kB
Buffers: 75100216 kB
Cached: 6990768 kB
SwapCached: 0 kB
Active: 69447392 kB
Inactive: 38154608 kB
Active(anon): 26890392 kB
Inactive(anon): 709920 kB
Active(file): 42557000 kB
Inactive(file): 37444688 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 180 kB
Writeback: 0 kB
AnonPages: 25511156 kB
Mapped: 35036 kB
Shmem: 2090184 kB
Slab: 1835540 kB
SReclaimable: 1684472 kB
SUnreclaim: 151068 kB
KernelStack: 15536 kB
PageTables: 59556 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 65877788 kB
Committed_AS: 5750772 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 551220 kB
VmallocChunk: 34358888444 kB
HardwareCorrupted: 0 kB
AnonHugePages: 21006336 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 294840 kB
DirectMap2M: 11159552 kB
DirectMap1G: 124780544 kB
cat /proc/buddyinfo
Node 0, zone DMA 1 0 1 0 2 1 1 0 1 1 3
Node 0, zone DMA32 785 898 869 489 260 227 442 316 142 0 0
Node 0, zone Normal 71568 1575732 446338 39 9 0 0 0 0 0 0
sar -B -r 1
Linux 3.10.0-514.el7.x86_64 (PZ-R-01) 02/09/2018 _x86_64_ (32 CPU)
02:13:57 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
02:13:58 PM 139388.00 48.00 4622.00 0.00 399474.00 0.00 17844.00 17908.00 100.36
02:13:57 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
02:13:58 PM 20986792 110768788 84.07 75421044 7021576 5743480 4.36 69832168 38119340 200
It seems that free memory pages is enough, why kswapd still reclaim pages?
// 2018/3/13 updated
It can use the almost full memory to cache disk data when updated linux kernel from 3.10 to 4.4, so it must be caused by kswapd behavior，and linux kernel 3.11 had improved the page reclaim behaviour.
see details:
Linux_3.11#Memory_management