Azure - Website - WebJob - Active FTP Download - azure

I am working with Windows Azure Websites and Web Jobs.
I have a console application that I use to download an FTP file nightly. They recently switch from passive to active FTP. I do not have any control over this.
The code attached was working for passive and works for active on my computer. However, it does not work when I add it to a webjob on Azure.
In this code I am able to get the content length, so I am logging in correctly and I have the correct URL.
Dim request As FtpWebRequest = DirectCast(FtpWebRequest.Create(strTempFTPUrl), FtpWebRequest)
request.Method = WebRequestMethods.Ftp.GetFileSize
Dim nc As New NetworkCredential(FTPUserName, FTPPassword)
request.Credentials = nc
request.UseBinary = True
request.UsePassive = False
request.KeepAlive = True
request.Proxy = Nothing
' Get the result (size)
Dim resp As FtpWebResponse = DirectCast(request.GetResponse(), FtpWebResponse)
Dim contLen As Int64 = resp.ContentLength
' and now download the file
request = DirectCast(FtpWebRequest.Create(strTempFTPUrl), FtpWebRequest)
request.Method = WebRequestMethods.Ftp.DownloadFile
request.Credentials = nc
request.UseBinary = True
request.UsePassive = False
request.KeepAlive = True
request.Proxy = Nothing
resp = DirectCast(request.GetResponse(), FtpWebResponse)
The error that I receive is this:
The underlying connection was closed: An unexpected error occurred on a receive. This happens on the second "resp = DirectCast(request.GetResponse(), FtpWebResponse)"
Does anyone have any suggestions on what I can do?
Edit: This is not a VM so as far as I know I do not have control over the firewall. This is a standard website.
Thank you very much!

I was with this same problem, I was able to solve by increasing the connection limit per point. By default it comes set to 2 I increased to 10
req.ServicePoint.ConnectionLimit = 10;
If you have timeout problem, also change the properties timeout and readwritetimeout.
Below is the link for a case similar to ours.
How can I programmatically remove the 2 connection limit in WebClient

Related

Need to set JMS_IBM_LAST_MSG_IN_GROUP property to true for IBM MQ testing using JMeter

I am testing IBM MQ using JMeter and able to establish the connection with queue to send requests over it. However, I need to set "JMS_IBM_LAST_MSG_IN_GROUP" property as true for one of the message but unable to do so. I am using below piece of code while sending request or trying to set the property to true but it remains set to it's default value i.e. false when I am checking in backend. Any clue what I am missing here.
Note: Connection is being established in another sampler and making use of that connection here. This code is working fine to send any request, just that property is not getting set to true.
import java.time.Instant
import com.ibm.msg.client.jms.JmsConstants
def sess = System.getProperties().get("Session")
def destination = System.getProperties().get("Destination")
def producer = sess.createProducer(destination)
def rnd = new Random(System.currentTimeMillis())
def payload = String.format('${groupid}|${sequencenumber}|rest of the payload|')
def msg = sess.createTextMessage(payload)
println('Payload --> ' + payload)
msg.setBooleanProperty(JmsConstants.JMS_IBM_LAST_MSG_IN_GROUP,true)
def start = Instant.now()
producer.send(msg)
def stop = Instant.now()
producer.close()
SampleResult.setResponseData(msg.toString())
SampleResult.setDataType( org.apache.jmeter.samplers.SampleResult.TEXT)
SampleResult.setLatency( stop.toEpochMilli() - start.toEpochMilli())
Your code does not include anything to set the Group ID or sequence number. I assume we have all the relevant code shown, in which case, I think you are missing code something along these lines:
msg.setStringProperty("JMSXGroupID", groupid);
msg.setIntProperty("JMSXGroupSeq", sequencenumber);
As per JMS_IBM_LAST_MSG_IN_GROUP chapter of IBMMQ documentation
This property is ignored in the publish/subscribe domain and is not relevant when an application connects to a service integration bus.
In general it's not necessary to use this property, you can come up with your own custom one, i.e. for the producer:
msg.setBooleanProperty("X_CUSTOM_PROPERTY_LAST_MESSAGE",true)
and for the consumer:
msg.getBooleanProperty("X_CUSTOM_PROPERTY_LAST_MESSAGE")
More information: IBM MQ testing with JMeter - Learn How
Sharing as it might help someone else. Was able to set the property to true by making below changes, rest of the code is same as mentioned in original question
import com.ibm.msg.client.wmq.WMQConstants
def gid=String.format('${groupid}')
def msg = sess.createTextMessage()
println('Payload --> ' + payload)
msg.setStringProperty(WMQConstants.JMSX_GROUPID,gid)
msg.setBooleanProperty(WMQConstants.JMS_IBM_LAST_MSG_IN_GROUP,true)
msg.text=payload

Flask sessions: PERMANENT_SESSION_LIFETIME not being set correctly

So I am using a flask session. I am using the filesystem type so that I can store more session data than I otherwise could. My users want to keep session timeouts long, and the site uses minimal server storage, so this is fine. However, when I try to set the timeout session to 24 hours below, it still times out after 30 minutes.
application = Flask(__name__)
SECRET_KEY = os.urandom(32)
application.config['SESSION_PERMANENT'] = True
application.config['SESSION_TYPE'] = 'filesystem'
application.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours = 24)
application.config['SECRET_KEY'] = SECRET_KEY
Session(application)
#application.before_request
def make_session_permanent():
session.permanent = True
application.permanent_session_lifetime = timedelta(hours = 24)
What am I doing wrong here?
I believe I have found the issue, which was in the line : SECRET_KEY = os.urandom(32). Every time the app restarted (idle window in browser reloaded), session data was erased as a new secret key was generated, meaning the browser could not find the cookies it needed. I generated one key, externally, and then hardcoded this value into my code so it is the same every time.
I would change the line with session.permanent =true to session.modified = True and see if it works then.

Node-RED + DB2 - msg : string[18] "No response object"

So, I'm a beginner in Node-RED and need to make a simple API with DB2 queries through flows. I'm using node-red-contrib-db2 to accomplish that. The thing is, I managed to get the results to several payloads to the debugger node, either triggered by timestamp or HTTP Request. However, I can't get these results on HTTP Reply and can't find the reason. Is it a problem with the db2 plugin or just me?
Exported nodes below:
[{"id":"96197abb.fd4098","type":"http in","z":"b4aa8db5.217028","name":"","url":"/wastes","method":"get","upload":false,"swaggerDoc":"","x":150,"y":140,"wires":[["9affb306.caf7e"]]},{"id":"bda39d37.edb418","type":"http response","z":"b4aa8db5.217028","name":"","statusCode":"200","headers":{},"x":940,"y":100,"wires":[]},{"id":"41708443.e4670c","type":"inject","z":"b4aa8db5.217028","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":220,"y":40,"wires":[["22a6e217.ead65e"]]},{"id":"9d1e6783.eb246","type":"ibmdb","z":"b4aa8db5.217028","mydb":"3a218407.1cca74","name":"IOCDATA","x":560,"y":40,"wires":[["80e51c1b.23b378"],[]]},{"id":"80e51c1b.23b378","type":"debug","z":"b4aa8db5.217028","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":730,"y":40,"wires":[]},{"id":"22a6e217.ead65e","type":"function","z":"b4aa8db5.217028","name":"SQL Query","func":"msg.database = \"iocdata\";\nmsg.payload = \"select * from viseu.waste_view\";\nreturn msg;","outputs":1,"noerr":0,"x":390,"y":40,"wires":[["9d1e6783.eb246"]]},{"id":"4a6bd014.f39868","type":"ibmdb","z":"b4aa8db5.217028","mydb":"3a218407.1cca74","name":"IOCDATA","x":500,"y":140,"wires":[["bda39d37.edb418","74e28d3e.039be4"],[]]},{"id":"9affb306.caf7e","type":"function","z":"b4aa8db5.217028","name":"SQL Query","func":"msg.database = \"iocdata\";\nmsg.payload = \"select * from viseu.waste_view where id = 1\";\nreturn msg;","outputs":1,"noerr":0,"x":330,"y":140,"wires":[["4a6bd014.f39868"]]},{"id":"74e28d3e.039be4","type":"debug","z":"b4aa8db5.217028","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":950,"y":180,"wires":[]},{"id":"3a218407.1cca74","type":"IbmDBdatabase","z":"","host":"10.102.0.62","port":"50002","db":"iocdata"}]
This is an issue with the ibmdb node you are using - it is not reusing the received message when it sends its results. That means the msg.req and msg.res properties provided by the HTTP In node are not set on the message by the time it reaches the HTTP Response node. This means the response node doesn't not what request to respond to.
To work around the issue, one approach, which isn't ideal, is to store msg.req and msg.res in flow context using a Change node before the ibmdb node, and then copy them back onto the msg after the ibmdb node. This isn't ideal because it can only handle one request at a time.
It would be best to raise an issue against the ibmdb node.
I managed to reach success in my flow, at the cost of many workarounds and variable juggling. But it IS working now. Select count + select rows + join rows where msg.complete is set when count value is reached. Here is the code:
[{"id":"96197abb.fd4098","type":"http in","z":"b4aa8db5.217028","name":"","url":"/wastes","method":"get","upload":false,"swaggerDoc":"","x":90,"y":140,"wires":[["d5f42a96.83f688"]]},{"id":"bda39d37.edb418","type":"http response","z":"b4aa8db5.217028","name":"","statusCode":"200","headers":{},"x":980,"y":260,"wires":[]},{"id":"4a6bd014.f39868","type":"ibmdb","z":"b4aa8db5.217028","mydb":"3a218407.1cca74","name":"SELECT waste_view","x":360,"y":200,"wires":[["35f99a5a.c7f87e"],[]]},{"id":"9affb306.caf7e","type":"function","z":"b4aa8db5.217028","name":"SQL Query","func":"msg.database = \"iocdata\";\nmsg.payload = \"select count(*) from viseu.waste_view\";\n\nreturn msg;","outputs":1,"noerr":0,"x":170,"y":200,"wires":[["4a6bd014.f39868"]]},{"id":"74e28d3e.039be4","type":"debug","z":"b4aa8db5.217028","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":890,"y":380,"wires":[]},{"id":"d5f42a96.83f688","type":"change","z":"b4aa8db5.217028","name":"","rules":[{"t":"set","p":"req","pt":"flow","to":"req","tot":"msg"},{"t":"set","p":"res","pt":"flow","to":"res","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":260,"y":140,"wires":[["9affb306.caf7e"]]},{"id":"c3ebb136.aa8988","type":"change","z":"b4aa8db5.217028","name":"","rules":[{"t":"set","p":"req","pt":"msg","to":"req","tot":"flow"},{"t":"set","p":"res","pt":"msg","to":"res","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":800,"y":260,"wires":[["bda39d37.edb418"]]},{"id":"ca59ece2.844b3","type":"join","z":"b4aa8db5.217028","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":630,"y":260,"wires":[["c3ebb136.aa8988","74e28d3e.039be4"]]},{"id":"35f99a5a.c7f87e","type":"function","z":"b4aa8db5.217028","name":"SQL Query","func":"msg.rowcount = msg.payload[1];\nmsg.database = \"iocdata\";\nmsg.payload = \"select * from viseu.waste_view\";// fetch first \" + msg.count[1] + \" rows only\";\n\nreturn msg;","outputs":1,"noerr":0,"x":550,"y":200,"wires":[["327a8ae.a8ce2f6"]]},{"id":"2666e2ba.41dc8e","type":"ibmdb","z":"b4aa8db5.217028","mydb":"3a218407.1cca74","name":"SELECT waste_view","x":800,"y":200,"wires":[["9008e06f.bf6d7"],[]]},{"id":"ec61a7f3.68cf8","type":"debug","z":"b4aa8db5.217028","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"count","x":650,"y":320,"wires":[]},{"id":"327a8ae.a8ce2f6","type":"change","z":"b4aa8db5.217028","name":"","rules":[{"t":"set","p":"rowcount","pt":"flow","to":"rowcount","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":670,"y":140,"wires":[["2666e2ba.41dc8e"]]},{"id":"90204f2d.8bafe8","type":"change","z":"b4aa8db5.217028","name":"","rules":[{"t":"set","p":"rowcount","pt":"msg","to":"rowcount","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":320,"wires":[["6888cd0d.d00064"]]},{"id":"9008e06f.bf6d7","type":"counter","z":"b4aa8db5.217028","name":"","init":"0","step":"1","lower":"","upper":"","mode":"increment","outputs":"1","x":220,"y":260,"wires":[["90204f2d.8bafe8"]]},{"id":"6888cd0d.d00064","type":"function","z":"b4aa8db5.217028","name":"if rowcount === count","func":"if (msg.count === msg.rowcount) {\n msg.complete = true;\n}\n\nreturn msg;","outputs":1,"noerr":0,"x":440,"y":260,"wires":[["ca59ece2.844b3","ec61a7f3.68cf8","a63f6ad6.26f08"]]},{"id":"a63f6ad6.26f08","type":"debug","z":"b4aa8db5.217028","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"rowcount","x":660,"y":380,"wires":[]},{"id":"3a218407.1cca74","type":"IbmDBdatabase","z":"","host":"10.102.0.62","port":"50002","db":"iocdata"}]
EDIT 21/02/2018: the previous solution is not very good, because the counter saves its value mysteriously and I can't reset it as I wanted it. That makes the counter surpass the wished rowcount value. So, I had to make my own counter in a function node. New code below:
[{"id":"96197abb.fd4098","type":"http in","z":"b4aa8db5.217028","name":"","url":"/wastes","method":"get","upload":false,"swaggerDoc":"","x":90,"y":60,"wires":[["d5f42a96.83f688"]]},{"id":"bda39d37.edb418","type":"http response","z":"b4aa8db5.217028","name":"","statusCode":"200","headers":{},"x":720,"y":220,"wires":[]},{"id":"4a6bd014.f39868","type":"ibmdb","z":"b4aa8db5.217028","mydb":"3a218407.1cca74","name":"SELECT waste_view","x":740,"y":60,"wires":[["35f99a5a.c7f87e"],[]]},{"id":"9affb306.caf7e","type":"function","z":"b4aa8db5.217028","name":"SQL Query","func":"msg.database = \"iocdata\";\nmsg.payload = \"select count(*) from viseu.waste_view\";\n\nreturn msg;","outputs":1,"noerr":0,"x":530,"y":60,"wires":[["4a6bd014.f39868"]]},{"id":"74e28d3e.039be4","type":"debug","z":"b4aa8db5.217028","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":550,"y":280,"wires":[]},{"id":"d5f42a96.83f688","type":"change","z":"b4aa8db5.217028","name":"save req and res","rules":[{"t":"set","p":"req","pt":"flow","to":"req","tot":"msg"},{"t":"set","p":"res","pt":"flow","to":"res","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":60,"wires":[["9affb306.caf7e"]]},{"id":"ca59ece2.844b3","type":"join","z":"b4aa8db5.217028","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"msg.count","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":390,"y":220,"wires":[["74e28d3e.039be4","c3ebb136.aa8988"]]},{"id":"35f99a5a.c7f87e","type":"function","z":"b4aa8db5.217028","name":"SQL Query","func":"msg.rowcount = msg.payload[1];\nmsg.database = \"iocdata\";\nmsg.payload = \"select * from viseu.waste_view\";\n\nreturn msg;","outputs":1,"noerr":0,"x":950,"y":60,"wires":[["327a8ae.a8ce2f6"]]},{"id":"2666e2ba.41dc8e","type":"ibmdb","z":"b4aa8db5.217028","mydb":"3a218407.1cca74","name":"SELECT waste_view","x":380,"y":140,"wires":[["90204f2d.8bafe8"],[]]},{"id":"327a8ae.a8ce2f6","type":"change","z":"b4aa8db5.217028","name":"save rowcount","rules":[{"t":"set","p":"rowcount","pt":"flow","to":"rowcount","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":160,"y":140,"wires":[["2666e2ba.41dc8e"]]},{"id":"90204f2d.8bafe8","type":"change","z":"b4aa8db5.217028","name":"get rowcount and count","rules":[{"t":"set","p":"rowcount","pt":"msg","to":"rowcount","tot":"flow"},{"t":"set","p":"count","pt":"msg","to":"count","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":630,"y":140,"wires":[["6888cd0d.d00064"]]},{"id":"6888cd0d.d00064","type":"function","z":"b4aa8db5.217028","name":"if count === rowcount","func":"//fix: msg.count ultrapassa msg.rowcount\nmsg.count = msg.count+1 || 1;\n\nif (msg.count === msg.rowcount) {\n msg.complete = true;\n msg.count = 0;\n}\n\nreturn msg;","outputs":1,"noerr":0,"x":880,"y":140,"wires":[["82ecfa98.9473d8"]]},{"id":"c3ebb136.aa8988","type":"change","z":"b4aa8db5.217028","name":"get req, res","rules":[{"t":"set","p":"req","pt":"msg","to":"req","tot":"flow"},{"t":"set","p":"res","pt":"msg","to":"res","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":550,"y":220,"wires":[["bda39d37.edb418"]]},{"id":"82ecfa98.9473d8","type":"change","z":"b4aa8db5.217028","name":"save count","rules":[{"t":"set","p":"count","pt":"flow","to":"count","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":210,"y":220,"wires":[["ca59ece2.844b3"]]},{"id":"3a218407.1cca74","type":"IbmDBdatabase","z":"","host":"10.102.0.69","port":"50002","db":"iocdata"}]

How to check if PySerial port.write() has written anything?

I have posted several question regarding a similar problem, here and here but after thinking and troubleshooting over the weekend, I'd like to explore more possible cause, but lack the knowledge.
How should I check ion my current code is writing anything to the sensor I have connected via USB-RS232. Here is my code.
port = serial.Serial("/dev/ttyUSB0", baudrate=9600, timeout=20, bytesize=8, rtscts =1,dsrdtr=1)
f_w = open('/home/ryan/python_serial_output.txt','r+')
f_o = open('/home/ryan/python_serial_parse.txt','w')
port.send_break()
sys_reply = port.read(100000)
sys_reply_str = sys_reply.decode('cp437')
print(sys_reply_str)
sys_reply_str_haha = sys_reply_str.replace("\r","")
sys_reply_str_haha = sys_reply_str_haha.replace("\n","")
i = list(sys_reply_str_haha)
if str(i[-1]) == '>':
port.reset_input_buffer()
print("ip_b_reset")
port.reset_output_buffer()
print("op_b_reset")
print(port.writable())
ip = 'CR1'
ip_en = ip.encode('cp437')
port.write(ip_en)
read_syscheck = port.read(1000)
read_syscheck_str = read_syscheck.decode('cp437')
print(read_syscheck_str)
With the same decoding format I tried writing to the port and should receive back a reply
>CR1
*[Parameters set to FACTORY defaults]*
Instead, I got
Sensor 2009
All rights reserved.
Firmware Version: 34.11
>
ip_b_reset
op_b_reset
True
CR1
7F7FD30000000030C6A87F9978B9302....
and that is why I conclude I might not be writing to the sensor.
PS: The data stream 7F7FD30000000030C6A87F9978B9302.... is always present and would appear when timed out. Hence I am unsure if this is the start of an actual reading? Or just a random readout.

Is it possible to specify an id when creating an issue on GitLab?

I intend to transfer issues from Redmine to GitLab using this script
https://github.com/sdslabs/redmine-to-gitlab/blob/master/issue-tranfer.py
It works, but I would like to keep the issues ids during the transition. By default GitLab just starts from #1 and increases. I tried adding "newissue['iid']=issue['id']" and variations to the parameters, but apparently GitLab simply does not permit assigning an id. Anyone knows if there's a way?
"issue" is the data acquired from redmine:
newissue = {}
newissue['id'] = pro['id']
newissue['title'] = issue['subject']
newissue['description'] = issue["description"]
if 'assigned_to' in issue:
auser = con.finduserbyname(issue['assigned_to']['name'])
if(auser):
newissue['assignee_id'] = auser['id']
print newissue
if ('fixed_version' in issue):
newissue['milestone_id'] = issue['fixed_version']['id']
newiss = post('/projects/' + str(pro['id']) + '/issues', newissue)
and this is the "post" function
def post( url, load = {}):
load['private_token'] = conf.token
r = requests.post(conf.base_url + url, params = load, verify = conf.sslverify)
return r.json()
The API does not allow you to specify an issue ID at creation time. The ID is intended to be sequential. The only way you could potentially accomplish this task is to interact with the database directly. If you choose this route I caution you to be extremely careful, and have backups.

Resources