I have a base64 encoded JWT that decodes on jwt.io and on base64decode.org but does not decode using base64 command on my OSX not my Linux machine (running echo {the string} | grep base64 --decode. I have no idea why, here is the string:
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2NvdW50IjoiNTgzZDQzZTItY2Q2ZC00YzBjLTlkNTItMmE4ZmNlY2Y2ZTVmIiwiYWNjb3VudFJvbGVzIjpbIk9PU19QRVJTT05fUkVNQVJLU19FRElUT1IiLCJBV1MtQVJUQVBQLUFETUlOIiwiT09TX1BFUlNPTl9ET0NVTUVOVFNfUkVBREVSX05PTl9NRURJQ0FMIiwiQVdTLVZFU1NFTExPQURJTkctQURNSU4iLCJPT1NfUEVSU09OX1JFTUFSS1NfUkVBREVSX1NBTEFSWSIsIlZQTlVTRVI5IiwiT09TX0NIRUNLTElTVF9BRE1JTiIsIkFXUy1QUk9ELUFETUlOIiwiQ09OQ09VUlNFLkNJIiwiT09TX1BPUlRfQ09NTUVOVFNfRURJVE9SIiwiT09TX1BFUlNPTl9FRElUT1JfU0VSVklDRSIsIk9PU19WRVNTRUxfUkVTUE9OU0lCSUxJVElFU19FRElUT1IiLCJNQU5BR0VEIERBVEFCQVNFUyIsIlZQTlVTRVIxMCIsIk9PU19QT1JUX01ZX0NPTU1FTlRTX0RFTEVUT1IiLCJPT1NfV09SS0ZMT1dfVVNFUiIsIk9PU19QRVJTT05fRURJVE9SIiwiQVdTLVNBTkRCT1gtQURNSU4iLCJPT1NfU1VQRVJfVVNFUiIsIlZQTlVTRVI1IiwiT09TX1ZFU1NFTF9JVElORVJBUllfUkVBREVSIiwiT09TX1BFUlNPTl9SRUFERVJfQkFDS0dST1VORCIsIk9PU19QRVJTT05fTUVESUNBTF9FRElUT1IiLCJPT1NfUEVSU09OX0RFTEVUT1IiLCJPT1NfSU5TUEVDVElPTl9TQ0hFRFVMRV9ERUxFVE9SIiwiT09TX1BFUlNPTl9SRUFERVJfU0VSVklDRSIsIk9PU19QRVJTT05fRE9DVU1FTlRTX0VESVRPUiIsIk9PU19QRVJTT05fRE9DVU1FTlRTX0RFTEVUT1IiLCJWUE5VU0VSMiIsIk9PU19QT1JUX0NPTU1FTlRTX0RFTEVUT1IiLCJPT1NfUEVSU09OX1JFTUFSS1NfUkVBREVSX1BFUlNPTkFMIiwiT09TX1BPUlRfTVlfQ09NTUVOVFNfRURJVE9SIiwiT09TX1RFTVBMQVRFX0xJQlJBUllfRURJVE9SIiwiQVdTLVRFU1QtQURNSU5AOTBQT0UuSU8iLCJPT1NfUEVSU09OX0RPQ1VNRU5UU19ET1dOTE9BREVSX01FRElDQUwiLCJJTlNQRUNUSU9OX1RFTVBMQVRFIiwiVlBOVVNFUjgiLCJPT1NfUEVSU09OX1JBTktfQVZBSUxBQklMSVRZX0VESVRPUiIsIlZQTlVTRVI0IiwiVlBOVVNFUjEiLCJPT1NfUEVSU09OX01FRElDQUxfREVMRVRPUiIsIlZQTlVTRVIzIiwiQVdTLUlBTS1BRE1JTiIsIkFXUy1BUlRBUFAtUFJPRC1BRE1JTiIsIk9PU19GTEVFVF9QRVJGT1JNQU5DRSIsIk9PU19JTlNQRUNUSU9OX1NDSEVEVUxFX1JFQURFUiIsIk9PU19QRVJTT05fRE9DVU1FTlRTX0RPV05MT0FERVJfTk9OX01FRElDQUwiLCJPT1NfUEVSU09OX0RPQ1VNRU5UU19SRUFERVJfTUVESUNBTCIsIk9PU19QT1JUX0lORk9fUkVBREVSIiwiVlBOVVNFUjYiLCJPT1NfUEVSU09OX1JFTUFSS1NfUkVBREVSX01FRElDQUwiLCJBV1MtREVWLUFETUlOIiwiQVdTLVRPT0xTLUFETUlOIiwiQVdTLUJJTExJTkctQURNSU4iLCJJTlNQRUNUSU9OX1RFTVBMQVRFX0lURU0iLCJPT1NfUEVSU09OX1JFTUFSS1NfREVMRVRPUiIsIk9PU19QRVJTT05fUkVBREVSX1NVTU1BUlkiLCJPT1NfUEVSU09OX1JFQURFUl9QSFlTSUNBTCIsIk9PU19XT1JLRkxPV19FRElUT1IiLCJPT1NfVEVNUExBVEVfTElCUkFSWV9SRUFERVIiLCJBUFBST0FDSF9JTkZPX0VESVRPUiIsIk9PU19URU1QTEFURV9MSUJSQVJZX0RFTEVUT1IiLCJBV1MtU1RHLUFETUlOIiwiT09TX1BFUlNPTl9SRU1BUktTX1JFQURFUl9UUkFWRUwiLCJPT1NfTUFOTklOR19BR0VOVF9WQVJOQSIsIk9PU19QRVJTT05fUkVBREVSX0NPTlRBQ1QiLCJPT1NfUEVSU09OX1JFQURFUl9ORVhUT0ZLSU4iLCJPT1NfSU5TUEVDVElPTl9TQ0hFRFVMRV9FRElUT1IiLCJFVkVSWU9ORSIsIk9PU19QRVJTT05fUkVBREVSX0ZBTUlMWSIsIk9PU19QRVJTT05fUkVNQVJLU19SRUFERVJfRElTQ0lQTElOQVJZIl0sImFwcGxpY2F0aW9uIjoiT09TIiwiY29udGV4dCI6Ik1BTk5JTkdfT0ZGSUNFOjEyOWQxNjAzLWFhZTktNDQ1NC04OWRjLWI4NzFhMGNmNGJhMCIsImV4cCI6MTU0MTc3Njg3OCwiaWF0IjoxNTQxNzc2Mjc4LCJpc3MiOiIiLCJwZXJtaXNzaW9ucyI6WyJJTlNQRUNUSU9OX1NDSEVEVUxFOkRMUlVDIiwiUEVSU09OX1JFTUFSS1NfU0FMQVJZOkNMUlVEIiwiUEVSU09OX0RPQ1VNRU5UU19DRVJUSUZJQ0FURVM6TFJDTlVEIiwiUEVSU09OOkxSQ1VEIiwiUEVSU09OX05FWFRPRktJTjpDTFJVRCIsIlBFUlNPTl9SRU1BUktTX01FRElDQUw6Q0xSVUQiLCJQRVJTT05fRE9DVU1FTlRTX0xJQ0VOU0U6TFJDTlVEIiwiUEVSU09OX0ZBTUlMWTpDTFJVRCIsIlBPUlRfSU5GT19BUFBST0FDSDpMUlVDIiwiSU5TUEVDVElPTl9URU1QTEFURTpMUlVDRCIsIklOU1BFQ1RJT05fVEVNUExBVEVfSVRFTTpMUlVDRCIsIlBMQVRGT1JNX1JFTUFSS1M6TFJVQ0QiLCJQTEFURk9STV9XT1JLRkxPV19ERUZJTklUSU9OOkNVIiwiVkVTU0VMX0lUSU5FUkFSWTpMUiIsIlBPUlRfQ09NTUVOVFM6TFJVQ0QiLCJQRVJTT05fUkFOS19BVkFJTEFCSUxJVFk6TFJDVUQiLCJQRVJTT05fQkFDS0dST1VORDpDTFJVRCIsIkZMRUVUX1BFUkZPUk1BTkNFOkxSIiwiUEVSU09OX1NFUlZJQ0U6TFJVQ0QiLCJQRVJTT05fQ09OVEFDVDpDTFJVRCIsIlBMQVRGT1JNX0RPQ1VNRU5UUzpMUlVDTkQiLCJQRVJTT05fUEhZU0lDQUw6Q0xSVUQiLCJQRVJTT05fUkVNQVJLU19QRVJTT05BTDpDTFJVRCIsIlBMQVRGT1JNX1dPUktGTE9XX0lOU1RBTkNFOkNVIiwiUEVSU09OX0RPQ1VNRU5UU19NRURJQ0FMOkxSVUNORCIsIlBFUlNPTl9SRU1BUktTX1RSQVZFTDpMUiIsIlBFUlNPTl9SRU1BUktTX0RJU0NJUExJTkFSWTpDTFJVRCIsIlBPUlRfTVlfQ09NTUVOVFM6TFJVQ0QiLCJQT1JUX0lORk86TFIiLCJWRVNTRUxfUkVTUE9OU0lCSUxJVElFUzpMUlVDIl0sInVzZXJJZCI6ImYxYzU3ZTRiLWEwZmEtNGJhZS04NWM4LTMzMzE1YWE4ZjJlZCJ9.NJ9jflW7Hk_RLL8X7olwvxeSl6bdeO783_iOkLegBcocZCP-0szh3kZ45gG3WAGN6O2KERuyg7lVGz1b5PT7CpaOyBGTGMYZDN6H6mqqs4EUidS10OV7191rYxLy0xh5axdw4O9Oci3fUL8U8ueO0AUNVomb_Tdw-3Ncgi7e35wmxv-PZiPe6fs_1aldy3BCCs4nZxV6cu8DeOmBRYuSC8qzYwmwZvWAlj_JAOOOT_1xvG8zQ6YUGPrNreADKZhVnbksjfGBee3fE3GjcFK8Sd1dbwap2hcw0n-FX-2PPK2IqLLeEdrOK1KcaxwHSfCupPGvXQa9ga03GMRhyd95Q9uZ6nSt2UijMEglCCQzSMKMQykriBsAD3_gQ375UnrXwzFdo6qfBWrNiBOuQiHN2E5XqMpRk1GYBH1K1cf8wp1ninGX4kJP6jbIGkAwNlsE0sLeXifUV0SLBHwEEFreq6BjLyXrWnmhXUPMoQOa-07zwx5iypwSSSr_o6x2xsEjKtJWA6kau_uArEfS7fYpTqKrOpZhPMZyrk5LojLgAnsPEgoDmm0PxMDDPaDSExXGwwDDumOw5UDrgoSp6Holt0XJ9sV1N2Jy7uoKCT6268ovIHOwjsrAKugNJAfGRYuxAzT9Dr5R3wxxZxw571yLAxitA_5fZeejgqZqrZ1MaII
I think the issue is that you're trying to decode the full JWT in a single hit. The JWT specification says that a JWT is three . separated base64 encoded pieces.
If you split the string on the two dots then each of the three pieces do decode using standard command line tools. The earlier poster is correct that underscores and hyphens are not valid in pure base64 encoding, but there are variants and some do allow these characters so provided you are following a consistent strategy those characters are fine.
What you have is not properly encoded base64. You have periods, underscores and hyphens in there.
I was able to clean up your string
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9eyJhY2NvdW50IjoiNTgzZDQzZTItY2Q2ZC00YzBjLTlkNTItMmE4ZmNlY2Y2ZTVmIiwiYWNjb3VudFJvbGVzIjpbIk9PU19QRVJTT05fUkVNQVJLU19FRElUT1IiLCJBV1MtQVJUQVBQLUFETUlOIiwiT09TX1BFUlNPTl9ET0NVTUVOVFNfUkVBREVSX05PTl9NRURJQ0FMIiwiQVdTLVZFU1NFTExPQURJTkctQURNSU4iLCJPT1NfUEVSU09OX1JFTUFSS1NfUkVBREVSX1NBTEFSWSIsIlZQTlVTRVI5IiwiT09TX0NIRUNLTElTVF9BRE1JTiIsIkFXUy1QUk9ELUFETUlOIiwiQ09OQ09VUlNFLkNJIiwiT09TX1BPUlRfQ09NTUVOVFNfRURJVE9SIiwiT09TX1BFUlNPTl9FRElUT1JfU0VSVklDRSIsIk9PU19WRVNTRUxfUkVTUE9OU0lCSUxJVElFU19FRElUT1IiLCJNQU5BR0VEIERBVEFCQVNFUyIsIlZQTlVTRVIxMCIsIk9PU19QT1JUX01ZX0NPTU1FTlRTX0RFTEVUT1IiLCJPT1NfV09SS0ZMT1dfVVNFUiIsIk9PU19QRVJTT05fRURJVE9SIiwiQVdTLVNBTkRCT1gtQURNSU4iLCJPT1NfU1VQRVJfVVNFUiIsIlZQTlVTRVI1IiwiT09TX1ZFU1NFTF9JVElORVJBUllfUkVBREVSIiwiT09TX1BFUlNPTl9SRUFERVJfQkFDS0dST1VORCIsIk9PU19QRVJTT05fTUVESUNBTF9FRElUT1IiLCJPT1NfUEVSU09OX0RFTEVUT1IiLCJPT1NfSU5TUEVDVElPTl9TQ0hFRFVMRV9ERUxFVE9SIiwiT09TX1BFUlNPTl9SRUFERVJfU0VSVklDRSIsIk9PU19QRVJTT05fRE9DVU1FTlRTX0VESVRPUiIsIk9PU19QRVJTT05fRE9DVU1FTlRTX0RFTEVUT1IiLCJWUE5VU0VSMiIsIk9PU19QT1JUX0NPTU1FTlRTX0RFTEVUT1IiLCJPT1NfUEVSU09OX1JFTUFSS1NfUkVBREVSX1BFUlNPTkFMIiwiT09TX1BPUlRfTVlfQ09NTUVOVFNfRURJVE9SIiwiT09TX1RFTVBMQVRFX0xJQlJBUllfRURJVE9SIiwiQVdTLVRFU1QtQURNSU5AOTBQT0UuSU8iLCJPT1NfUEVSU09OX0RPQ1VNRU5UU19ET1dOTE9BREVSX01FRElDQUwiLCJJTlNQRUNUSU9OX1RFTVBMQVRFIiwiVlBOVVNFUjgiLCJPT1NfUEVSU09OX1JBTktfQVZBSUxBQklMSVRZX0VESVRPUiIsIlZQTlVTRVI0IiwiVlBOVVNFUjEiLCJPT1NfUEVSU09OX01FRElDQUxfREVMRVRPUiIsIlZQTlVTRVIzIiwiQVdTLUlBTS1BRE1JTiIsIkFXUy1BUlRBUFAtUFJPRC1BRE1JTiIsIk9PU19GTEVFVF9QRVJGT1JNQU5DRSIsIk9PU19JTlNQRUNUSU9OX1NDSEVEVUxFX1JFQURFUiIsIk9PU19QRVJTT05fRE9DVU1FTlRTX0RPV05MT0FERVJfTk9OX01FRElDQUwiLCJPT1NfUEVSU09OX0RPQ1VNRU5UU19SRUFERVJfTUVESUNBTCIsIk9PU19QT1JUX0lORk9fUkVBREVSIiwiVlBOVVNFUjYiLCJPT1NfUEVSU09OX1JFTUFSS1NfUkVBREVSX01FRElDQUwiLCJBV1MtREVWLUFETUlOIiwiQVdTLVRPT0xTLUFETUlOIiwiQVdTLUJJTExJTkctQURNSU4iLCJJTlNQRUNUSU9OX1RFTVBMQVRFX0lURU0iLCJPT1NfUEVSU09OX1JFTUFSS1NfREVMRVRPUiIsIk9PU19QRVJTT05fUkVBREVSX1NVTU1BUlkiLCJPT1NfUEVSU09OX1JFQURFUl9QSFlTSUNBTCIsIk9PU19XT1JLRkxPV19FRElUT1IiLCJPT1NfVEVNUExBVEVfTElCUkFSWV9SRUFERVIiLCJBUFBST0FDSF9JTkZPX0VESVRPUiIsIk9PU19URU1QTEFURV9MSUJSQVJZX0RFTEVUT1IiLCJBV1MtU1RHLUFETUlOIiwiT09TX1BFUlNPTl9SRU1BUktTX1JFQURFUl9UUkFWRUwiLCJPT1NfTUFOTklOR19BR0VOVF9WQVJOQSIsIk9PU19QRVJTT05fUkVBREVSX0NPTlRBQ1QiLCJPT1NfUEVSU09OX1JFQURFUl9ORVhUT0ZLSU4iLCJPT1NfSU5TUEVDVElPTl9TQ0hFRFVMRV9FRElUT1IiLCJFVkVSWU9ORSIsIk9PU19QRVJTT05fUkVBREVSX0ZBTUlMWSIsIk9PU19QRVJTT05fUkVNQVJLU19SRUFERVJfRElTQ0lQTElOQVJZIl0sImFwcGxpY2F0aW9uIjoiT09TIiwiY29udGV4dCI6Ik1BTk5JTkdfT0ZGSUNFOjEyOWQxNjAzLWFhZTktNDQ1NC04OWRjLWI4NzFhMGNmNGJhMCIsImV4cCI6MTU0MTc3Njg3OCwiaWF0IjoxNTQxNzc2Mjc4LCJpc3MiOiIiLCJwZXJtaXNzaW9ucyI6WyJJTlNQRUNUSU9OX1NDSEVEVUxFOkRMUlVDIiwiUEVSU09OX1JFTUFSS1NfU0FMQVJZOkNMUlVEIiwiUEVSU09OX0RPQ1VNRU5UU19DRVJUSUZJQ0FURVM6TFJDTlVEIiwiUEVSU09OOkxSQ1VEIiwiUEVSU09OX05FWFRPRktJTjpDTFJVRCIsIlBFUlNPTl9SRU1BUktTX01FRElDQUw6Q0xSVUQiLCJQRVJTT05fRE9DVU1FTlRTX0xJQ0VOU0U6TFJDTlVEIiwiUEVSU09OX0ZBTUlMWTpDTFJVRCIsIlBPUlRfSU5GT19BUFBST0FDSDpMUlVDIiwiSU5TUEVDVElPTl9URU1QTEFURTpMUlVDRCIsIklOU1BFQ1RJT05fVEVNUExBVEVfSVRFTTpMUlVDRCIsIlBMQVRGT1JNX1JFTUFSS1M6TFJVQ0QiLCJQTEFURk9STV9XT1JLRkxPV19ERUZJTklUSU9OOkNVIiwiVkVTU0VMX0lUSU5FUkFSWTpMUiIsIlBPUlRfQ09NTUVOVFM6TFJVQ0QiLCJQRVJTT05fUkFOS19BVkFJTEFCSUxJVFk6TFJDVUQiLCJQRVJTT05fQkFDS0dST1VORDpDTFJVRCIsIkZMRUVUX1BFUkZPUk1BTkNFOkxSIiwiUEVSU09OX1NFUlZJQ0U6TFJVQ0QiLCJQRVJTT05fQ09OVEFDVDpDTFJVRCIsIlBMQVRGT1JNX0RPQ1VNRU5UUzpMUlVDTkQiLCJQRVJTT05fUEhZU0lDQUw6Q0xSVUQiLCJQRVJTT05fUkVNQVJLU19QRVJTT05BTDpDTFJVRCIsIlBMQVRGT1JNX1dPUktGTE9XX0lOU1RBTkNFOkNVIiwiUEVSU09OX0RPQ1VNRU5UU19NRURJQ0FMOkxSVUNORCIsIlBFUlNPTl9SRU1BUktTX1RSQVZFTDpMUiIsIlBFUlNPTl9SRU1BUktTX0RJU0NJUExJTkFSWTpDTFJVRCIsIlBPUlRfTVlfQ09NTUVOVFM6TFJVQ0QiLCJQT1JUX0lORk86TFIiLCJWRVNTRUxfUkVTUE9OU0lCSUxJVElFUzpMUlVDIl0sInVzZXJJZCI6ImYxYzU3ZTRiLWEwZmEtNGJhZS04NWM4LTMzMzE1YWE4ZjJlZCJ9NJ9jflW7HkRLL8X7olwvxeSl6bdeO783iOkLegBcocZCP0szh3kZ45gG3WAGN6O2KERuyg7lVGz1b5PT7CpaOyBGTGMYZDN6H6mqqs4EUidS10OV7191rYxLy0xh5axdw4O9Oci3fUL8U8ueO0AUNVombTdw3Ncgi7e35wmxvPZiPe6fs1aldy3BCCs4nZxV6cu8DeOmBRYuSC8qzYwmwZvWAljJAOOOT1xvG8zQ6YUGPrNreADKZhVnbksjfGBee3fE3GjcFK8Sd1dbwap2hcw0nFX2PPK2IqLLeEdrOK1KcaxwHSfCupPGvXQa9ga03GMRhyd95Q9uZ6nSt2UijMEglCCQzSMKMQykriBsAD3gQ375UnrXwzFdo6qfBWrNiBOuQiHN2E5XqMpRk1GYBH1K1cf8wp1ninGX4kJP6jbIGkAwNlsE0sLeXifUV0SLBHwEEFreq6BjLyXrWnmhXUPMoQOa07zwx5iypwSSSro6x2xsEjKtJWA6kauuArEfS7fYpTqKrOpZhPMZyrk5LojLgAnsPEgoDmm0PxMDDPaDSExXGwwDDumOw5UDrgoSp6Holt0XJ9sV1N2Jy7uoKCT6268ovIHOwjsrAKugNJAfGRYuxAzT9Dr5R3wxxZxw571yLAxitA5fZeejgqZqrZ1MaII
That decodes fine in https://www.developertoolkits.com/base64/decoder
#Gary Lloyd is right in that because JWT format is <header>.<payload>.<signature> and . is not part of base64, that causes base64 command to error out with invalid input.
What's more is that JWT format for <header>, <payload>, and <signature> are base64URL which is slight modification of base64.
Another note is that if you try to decode <signature>, it inherently has all kinds of funny characters(maybe binary?) which will fail if you try decoding it.
All that said, try separating the <header> and <payload> and it should decode just fine.
Here's a simple bash script that you can add in your bashrc file if you really want to use cli.
#!/usr/bin/env bash
function base64jwt() {
read content
token_parts=($(echo "$content" | tr "." "\n"))
echo "[$(base64 -d <<< "$token_parts[1]"), $(base64 -d <<< "$token_parts[2]")]"
}
Note that I added the content in an array format [<header_content>, <payload_content>] so that you can easily pipe that to tool like jq for prettier json format.
e.g.
echo "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG5Eb2UiLCJpYXQiOjE1MTYyMzkwMjJ9.RW7PfdmXPU5baGCV2WIXosKpuahEMUUY4nJvEh7gLDo" \
| base64jwt \
| jq
Related
I have to do simple structured bulk load on my redis database. However there are also some UTF-8 characters and when I'm trying to load in data with them I am getting ERR Protocol error: expected '$', got ' ' . Loading in data without UTF-8 characters works just fine.
Data example of UTF-8 char that is causing the error :
*4\r\n$4\r\nHSET\r\n$6\r\nGrad_Ž\r\n$6\r\nalmada\r\n$1\r\n1\r\n
If I replace Ž with normal character like S for example it loads and causes no errors.
I have tried different commands to run it and I have tried changing bash locale.
Command I am using to run it :
echo -e "$(cat test.txt)" | redis-cli --pipe
Thanks in advance.
In your case:
$6\r\nGrad_Ž\r\n
6 means that there's 6 bytes value, however, Grad_Ž has more than 6 bytes. Because Ž is NOT an ascii character, and takes more than 1 byte.
You need to set the correct string length (in byte) for the value.
Is it possible to read the emails based on the Subject line and then get the base64 attachment or directly get the attachment ?
Server : Linux System
Your question seems to presuppose that there is a single attachment and that it can be reliably extracted. In the general case, an email message can have a basically infinite amount of attachments, and the encoding could be one out of several.
But if we assume that you are dealing with a single sender which consistently uses a static message template where the first base64 attachment is always going to be the one you want, something like
case $(formail -zcxSubject: <"$message") in
"Hello, here is your report for "*)
awk 'BEGIN { h=1 }
h { if ($0 ~ /^$/) h=0 ; next } # skip headers
/^Content-Disposition: attachment/ { a=1 } # find att
a && /^$/ { p=1; next }
p && /^$/ { exit }
p' "$message" |
base64 -d ;;
esac
This will extract the Subject: header and compare it to a glob pattern. I expect this is what you mean by "based on subject" -- if we find a matching subject header, examine this message, otherwise discard.
The crude Awk script attempts to isolate the base64 data and pass it to base64 -d for extraction. This contains a number of pesky and somewhat crude assumptions about the message format, and probably requires significant additional tweaking. Briefly, we skip the headers, then look for MIME headers identifying an attachment, and print that, skipping everything else in the message. If this header is missing, or identifies the wrong MIME part, you will get no results, or (worse) incorrect results. Also, the /^Content-Disposition:/ regex could theoretically match on a line which is not a MIME header, though this seems highly unlikely (but might actually happen if you are looking e.g. at a bounce message).
A more robust approach would involve a MIME extraction tool or perhaps a custom script to actually parse the MIME structure and extract the part you want. Without details about what exactly you need, I'm not able to provide that. (This would also allow you to use the sender's specified filename; the above script simply prints the decoded payload to standard output.)
Note also that formail has no idea about RFC2047 encoding, so if the subject is not plain ASCII, you have to specify the encoded form in the script.
I am using below nodejs package to generate some simple restful APIs.
https://github.com/restify/node-restify
But I got a strange char at the end of the response json.
The server side code is the same as the above link provides.
How am I able to remove the %?
zsh adds a % sign to show that it was a partial line and no end-of-line delimiter was encountered. More here
CURL requests reply JSONs, without any EOL delimiter, hence % sign. Run the same command from bash shell and no such behaviour will be seen.
PS: Postman is really good tool to test APIs
I need to extract the variable from a JSON encoded file and assign it to a variable in Bash.
excerpt...from file.json
"VariableA": "VariableA data",
"VariableB": [
"VariableB1",
"VariableB2",
"VariableB3",
"VariableB3"
],
I've gotten somewhere with this
variableA=$(fgrep -m 1 "VariableA" file.json )
but it returns the whole line. I just want the data
For the VariableB I need to replace the list with comma separated values.
I've looked at awk, sed, grep, regexpressions and really given the learning curve...need to know which one to use, or a better solution.
Thanks for your suggestions...but this is perfect
git://github.com/kristopolous/TickTick.git
You are better off using a JSON parser. There are many listed at http://json.org/ including two for the BASH shell.
http://kmkeen.com/jshon/
https://github.com/dominictarr/JSON.sh
There is powerful command-line JSON tool jq.
Extracting single value is easy:
variableA=$(jq .VariableA file.json)
For comma separated array contents try this
variableB=$(jq '.VariableB | #csv' file.json)
or
variableB=$(jq '.VariableB | .[]' file.json | tr '\n' ',' | head -c-1)
If you're open to using Perl they have a 'open()' function that will pipe a file with the json function 'to_json'. And if you want to extract json you can use the 'from_json' function. You can check it out here:
http://search.cpan.org/~rjbs/perl-5.16.0/lib/open.pm
http://metacpan.org/pod/JSON#to_json
http://metacpan.org/pod/JSON#from_json ( you might also try using decode json as well)
inside a webapplication i am processing requests to a url like
http://example.com/<website-base-url>
im am logging the raw GET parameter of the request in an uft8 database column and in filesystem. for a few chinese domains i get requests with a website-base-url parameter like
%C3%83%C2%A3%C3%82%C2%A5%C3%83%C2%A2%C3%82%C2%A4%C3%83%C2%A2%C3%82%C2%A7%C3%83%C2%A3%C3%82%C2%A5%C3%83%C2%A2%C3%82%C2%A4%C3%83%C2%A2%C3%82%C2%B4%C3%83%C2%A3%C3%82%C2%A8%C3%83%C2%A2%C3%82%C2%B4%C3%83%C2%A2%C3%82%C2%B4.cn
Decoding with urldecode returns
ã¥â¤â§ã¥â¤â´ã¨â´â´.cn
This does not seem to be the domain name the user wants to request.
I have tried urlencoding, base64, utf8 and combinations wihtout success.
Any suggestions how decode the given parameter to utf8?
URL percentage encodings simply encode the raw bytes. It does not give you any hint regarding the actual encoding of the text. If you do not know what encoding these bytes represent, all you can do is guess.
php > $d = urldecode('%C3%83%C2%A3%C3%82%C2%A5%C3%83%C2%A2%C3%82%C2%A4%C3%83%C2%A2%C3%82%C2%A7%C3%83%C2%A3%C3%82%C2%A5%C3%83%C2%A2%C3%82%C2%A4%C3%83%C2%A2%C3%82%C2%B4%C3%83%C2%A3%C3%82%C2%A8%C3%83%C2%A2%C3%82%C2%B4%C3%83%C2%A2%C3%82%C2%B4.cn');
php > echo $d;
ã¥â¤â§ã¥â¤â´ã¨â´â´.cn
php > echo iconv('BIG5', 'UTF-8', $d);
php > echo iconv('Shift-JIS', 'UTF-8', $d);
ﾃδ｣ﾃつ･ﾃδ｢ﾃつ､ﾃδ｢ﾃつｧﾃδ｣ﾃつ･ﾃδ｢ﾃつ､ﾃδ｢ﾃつｴﾃδ｣ﾃつｨﾃδ｢ﾃつｴﾃδ｢ﾃつｴ.cn
php > echo iconv('GB18030', 'UTF-8', $d);
脙拢脗楼脙垄脗陇脙垄脗搂脙拢脗楼脙垄脗陇脙垄脗麓脙拢脗篓脙垄脗麓脙垄脗麓.cn
GB18030 would seem to be the best candidate, but even that decoded string looks a bit too repetitive to be really useful Chinese.