Laravel Factory error Allowed memory size of 536870912 bytes exhausted (tried to allocate 262144 bytes) - laravel-7

I want to generate dummy data using factory with seeder so it will give me the error.
when I run this command given below:
php artisan db:seed
so here it's the error.
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 262144 bytes) in D:\xampp\htdocs\Bootstrap\vendor\laravel\framework\src\Illuminate\Database\Query\Grammars\Grammar.php on line 1120
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 262144 bytes) in Unknown on line 0
class DatabaseSeeder extends Seeder
{
public function run()
{
factory(User::class,5)->create()->each(function ($user){
$profile = factory(Profile::class)->make();
$user->profile()->save($profile);
$profile->each(function ($profiles){
$qualification =factory(Qualification::class,3)->make();
$experience =factory(Experience::class,3)->make();
$profiles->qualification()->saveMany($qualification);
$profiles->experience()->saveMany($experience);
});
});
}
}
For each User has-one Profile.
For each Profile Has-many (Qualification and Experience).
If we run this code given below:
class DatabaseSeeder extends Seeder
{
public function run()
{
DB::table('posts')->insertOrIgnore([
['id'=>1,'title'=>'admission','created_at'=>now(),'updated_at'=>now()],
['id'=>2,'title'=>'biology','created_at'=>now(),'updated_at'=>now()],
['id'=>3,'title'=>'mathematics','created_at'=>now(),'updated_at'=>now()],
['id'=>4,'title'=>'chemistry','created_at'=>now(),'updated_at'=>now()],
['id'=>5,'title'=>'physics','created_at'=>now(),'updated_at'=>now()],
['id'=>6,'title'=>'english','created_at'=>now(),'updated_at'=>now()],
['id'=>7,'title'=>'urdu','created_at'=>now(),'updated_at'=>now()],
]);
DB::table('provinces')->insertOrIgnore([
['id'=>1,'title'=>'punjab','created_at'=>now(),'updated_at'=>now()],
['id'=>2,'title'=>'sindh','created_at'=>now(),'updated_at'=>now()],
['id'=>3,'title'=>'nwfp','created_at'=>now(),'updated_at'=>now()],
['id'=>4,'title'=>'balochistan','created_at'=>now(),'updated_at'=>now()],
]);
}
}
using this command php artisan db:seed
then there is no error received.
please help me in using a laravel factory.

To fix that issue edit your php.ini.
Increase the memory limit by 512M >
; Maximum amount of memory a script may consume (128 MB)
; http://php.net/memory-limit
memory_limit=512M
or make it unlimited.(it depends on your server resources)
; Maximum amount of memory a script may consume (128 MB)
; http://php.net/memory-limit
memory_limit=512M

Related

What are the effective ways to work Node js with a large JSON file of 600 MB and more?

What are the effective ways to work Node js with a large JSON file of 600 MB and more?
My partner gives me from his REST API wery large JSON file. 600mb, 1000mb
Its structure is as follows
{ nameid1:[list id,....], nameid2:[list id,....], }
[list id,....] - An array with ID can be up to hundreds of millions of records.
Now to work with such files I use the following sequence of actions.
I save it to hard drive
With the sed command, from a single-line file, I make it multi-line
Example
exec (`sed -i 's /', '/', '\ n / g' file.json)
I work directly with the file using readline
I tried to use JSONStream but it causes FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
function getStream() {
let jsonData = __dirname + '/jsonlarge/file.json',
stream = fs.createReadStream(jsonData, {
encoding: 'utf8'
})
parser = JSONStream.parse('*');
stream.pipe(parser)
parser.on('data', (data) => {
console.log('received:', data);
});
}
Example structure json file
{"Work":"12122001","name":"Regist","world":[{"name":"000000","point":"rfg","Content":["3202b9a3fba","121323","2343454","45345543","354534534"]}, {"name":"000000","point":"rfg","Content":["3202b","121323","2343454","45345543","354534534"]}, {"name":"000000","point":"rfg","Content":["320","121323","2343454","45345543","354534534"]}]}
Maybe someone knows a faster way to work with such files.
Thanks

Too many open files in spark aborting spark job

In my application i am reading 40 GB text files that is totally spread across 188 files .
I split this files and create xml files per line in spark using pair rdd .
For 40 GB of input it will create many millions small xml files and this is my requirement.
All working fine but when spark saves files in S3 it throws error and job fails .
Here is the exception i get
Caused by: java.nio.file.FileSystemException:
/mnt/s3/emrfs-2408623010549537848/0000000000: Too many open files at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361) at
java.nio.file.Files.createFile(Files.java:632) at
com.amazon.ws.emr.hadoop.fs.files.TemporaryFiles.create(TemporaryFiles.java:70)
at
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.openNewPart(MultipartUploadOutputStream.java:493)
... 21 more
ApplicationMaster host: 10.97.57.198 ApplicationMaster RPC port: 0
queue: default start time: 1542344243252 final status: FAILED
tracking URL:
http://ip-10-97-57-234.tr-fr-nonprod.aws-int.thomsonreuters.com:20888/proxy/application_1542343091900_0001/
user: hadoop Exception in thread "main"
org.apache.spark.SparkException: Application
application_1542343091900_0001 finished with failed status
And this as well
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception:
Please reduce your request rate. (Service: Amazon S3; Status Code:
503; Error Code: SlowDown; Request ID: D33581CA9A799F64; S3 Extended
Request ID:
/SlEplo+lCKQRVVH+zHiop0oh8q8WqwnNykK3Ga6/VM2HENl/eKizbd1rg4vZD1BZIpp8lk6zwA=),
S3 Extended Request ID:
/SlEplo+lCKQRVVH+zHiop0oh8q8WqwnNykK3Ga6/VM2HENl/eKizbd1rg4vZD1BZIpp8lk6zwA=
Here is my code to do that .
object TestAudit {
def main(args: Array[String]) {
val inputPath = args(0)
val output = args(1)
val noOfHashPartitioner = args(2).toInt
//val conf = new SparkConf().setAppName("AuditXML").setMaster("local");
val conf = new SparkConf().setAppName("AuditXML")
val sc = new SparkContext(conf);
val input = sc.textFile(inputPath)
val pairedRDD = input.map(row => {
val split = row.split("\\|")
val fileName = split(0)
val fileContent = split(1)
(fileName, fileContent)
})
import org.apache.hadoop.io.NullWritable
import org.apache.spark.HashPartitioner
import org.apache.hadoop.mapred.lib.MultipleTextOutputFormat
class RddMultiTextOutputFormat extends MultipleTextOutputFormat[Any, Any] {
override def generateActualKey(key: Any, value: Any): Any = NullWritable.get()
override def generateFileNameForKeyValue(key: Any, value: Any, name: String): String = key.asInstanceOf[String]
}
pairedRDD.partitionBy(new HashPartitioner(10000)).saveAsHadoopFile("s3://a205381-tr-fr-development-us-east-1-trf-auditabilty//AUDITOUTPUT", classOf[String], classOf[String], classOf[RddMultiTextOutputFormat], classOf[GzipCodec])
}
}
Even i tried reducing no of HashPartitioner then also it does not work
Every process on Unix systems has a limitation of open files or number of file descriptors. As your data is large and partitions to subfile (in internal of Spark), your process meet the limitation and error.
You can increase the number of file descriptors for each user as following:
edit the file: /etc/security/limits.conf and add (or modify)
* hard nofile 500000
* soft nofile 500000
root hard nofile 500000
root soft nofile 500000
This will set the nofile (number of file descriptors) feature to 500000 for each user along with the root user.
After restarting the changes will be applied.
Also, someone can set the number of file descriptors for a special process, by setting the LimitNOFILE. For example, if you use yarn to run Spark jobs and the Yarn daemon will be started using systemd, you can add LimitNOFILE=128000 to Yarn systemd script(resource manager and nodemanager) to set Yarn process number of file descriptors to 128000.
related articles:
3 Methods to Change the Number of Open File Limit in Linux
Limits on the number of file descriptors

How to load very large csv files in nodejs?

I'm trying to load 2 big csv into nodejs, first one has a size of 257 597 ko and second one 104 330 ko. I'm using the filesystem (fs) and csv modules, here's my code :
fs.readFile('path/to/my/file.csv', (err, data) => {
if (err) console.err(err)
else {
csv.parse(data, (err, dataParsed) => {
if (err) console.err(err)
else {
myData = dataParsed
console.log('csv loaded')
}
})
}
})
And after ages (1-2 hours) it just crashes with this error message :
<--- Last few GCs --->
[1472:0000000000466170] 4366473 ms: Mark-sweep 3935.2 (4007.3) -> 3935.2 (4007.
3) MB, 5584.4 / 0.0 ms last resort GC in old space requested
[1472:0000000000466170] 4371668 ms: Mark-sweep 3935.2 (4007.3) -> 3935.2 (4007.
3) MB, 5194.3 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 000002BDF12254D9 <JSObject>
1: stringSlice(aka stringSlice) [buffer.js:590] [bytecode=000000810336DC91 o
ffset=94](this=000003512FC822D1 <undefined>,buf=0000007C81D768B9 <Uint8Array map
= 00000352A16C4D01>,encoding=000002BDF1235F21 <String[4]: utf8>,start=0,end=263
778854)
2: toString [buffer.js:664] [bytecode=000000810336D8D9 offset=148](this=0000
007C81D768B9 <Uint8Array map = 00000352A16C4D01>,encoding=000002BDF1...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memo
ry
1: node::DecodeWrite
2: node_module_register
3: v8::internal::FatalProcessOutOfMemory
4: v8::internal::FatalProcessOutOfMemory
5: v8::internal::Factory::NewRawTwoByteString
6: v8::internal::Factory::NewStringFromUtf8
7: v8::String::NewFromUtf8
8: std::vector<v8::CpuProfileDeoptFrame,std::allocator<v8::CpuProfileDeoptFrame
> >::vector<v8::CpuProfileDeoptFrame,std::allocator<v8::CpuProfileDeoptFrame> >
9: v8::internal::wasm::SignatureMap::Find
10: v8::internal::Builtins::CallableFor
11: v8::internal::Builtins::CallableFor
12: v8::internal::Builtins::CallableFor
13: 00000081634043C1
The biggest file is loaded but node runs out of memory for the other. It's probably easy to allocate more memory, but the main issue here is the loading time, it seems very long despite the size of files. So what is the correct way to do it? Python loads these csv really fast with pandas btw (3-5 seconds).
Stream works perfectly, it took only 3-5 seconds :
var csv = require('csv-parser')
var data = []
fs.createReadStream('path/to/my/data.csv')
.pipe(csv())
.on('data', function (row) {
data.push(row)
})
.on('end', function () {
console.log('Data loaded')
})
fs.readFile will load the entire file into memory, but fs.createReadStream will read the file in chunks of the size you specify.
This will prevent it from running out of memory
You may want to stream the CSV, instead of reading it all at once:
csv-parse has streaming support: http://csv.adaltas.com/parse/
or, you may want to take a look at csv-stream: https://www.npmjs.com/package/csv-stream

Nodejs read stdout of some process and do stuff

I want to read some stdout of some process with nodejs, in Windows.
So far i got this:
I simulate the process output with :
ping -t google.com > ping.txt
The ping output is dumped in the ping.txt file.
That file is watched in node for changes.
So, in nodejs i got this code:
var filename = 'ping.txt';
var fs = require("fs");
fs.watch(filename, function (event, filename) {
if (event == 'change') {
console.log('change', filename);
fs.stat(filename, function (err, stats) {
console.log('size ', stats.size);
if(stats.size > 500){
fs.truncate(filename,10,function(){
console.log("truncated");
});
}
})
}
});
But this is my nodejs output:
size 0
change ping.txt
size 192
change ping.txt
size 253
change ping.txt
size 375
change ping.txt
size 436
change ping.txt
size 559
truncated
change ping.txt
size 620
truncated
change ping.txt
size 743
truncated
change ping.txt
size 804
truncated
change ping.txt
size 926
truncated
The file never gets truncated.
I don't want to get a massive file size, because the real process is getting a lot of output.
So that's the main reason to try to get the file truncated.
But it's not working.
Can anyone give me a hand? This is my first nodejs experiment.
Later i'm planning to output that stdout process trought a websocket, but now i'm stucked with this problem.
Thanks in advance ! Best regards!
Edit 1:
The ping process is not the real one that i'm trying to read. The real process is a cryptocoin miner. And it's very resource hungry.
So that's because i took this aproach. To dump the process output in a file and read it with another process.
I'm not 100% happy to have nodejs managing that cryptocoin process, because i don't know if node can handle it.
If is there a better way to do this, i'm very happy to ear it. Thanks!
I did a small snippet using a npm package rotating-file-stream and it seems to work:
// stream.js
const rfs = require('rotating-file-stream');
const stream = rfs('ping.txt', {
size: '500B', // rotate every 500 Bytes written
});
const stdin = process.openStdin();
stdin.pipe(stream);
Run the command in terminal ping google.com | node stream.js

node js upload size unit is strange

I am working on upload module of my server and I set file uploads with multiparty. I am currently trying to limit the upload size simply i a doing something like this
req.on("data", function(dt) {
bytes += dt.length;
if (bytes > 2048) {
req.connection.destroy();
console.log("connection destroyed due to huge file size");
}
console.log(bytes);
});
I thought this length is in bytes and tried to limit it with 2mb
but i noticed this unit is a bit strange for testing i uploaded a 148 kb file but the length of the variable i created so far is 421 it is neither in bits nor bytes why it is so strange number? where do this extra ~300k come from?
Did you try filesystem module for checking size of the file?
E.g.
var fs = require("fs");
var stats = fs.statSync("myfile.txt");
var fileSizeInBytes = stats.size;

Resources