I'm trying to make an architecture for a data lake, I already generated my CSV, txt, and Avro files they are in an On-Premise machine and I want to upload them to Google Cloud Storage, but I see that I have to go through the public internet and I don't want to that.
What options do I have to make as safe as possible?
I was trying to make a Compute Engine environment to upload the files here through SFTP and then moved them to Cloud Storage, but that will make my cost to go up.
I think this the use case of Google VPC. The key benefit provided on the product description is:
For on-premises, you can share a connection between VPC and on-premises resources with all regions in a single VPC
If you look deeper into documentation you may find document Configuring Private Google Access for on-premises hosts. Some information maybe be found here as well. This might be a good option for you.
Related
I want to spike whether azure and the cloud is a good fit for us.
We have a website where users upload documents to our currently hosted website.
Every document has an equivalent record in a database.
I am using terraform to create the azure infrastructure.
What is my best way of migrating the documents from the local file path on the server to azure?
Should I be using file storage or blob storage. I am confused about the difference.
Is there anything in terraform that can help with this?
Based on your comments, I would recommend storing them in Blob Storage. This service is suited for storing and serving unstructured data like files and images. There are many other features like redundancy, archiving etc. that you may find useful in your scenario.
File Storage is more suitable in Lift-and-Shift kind of scenarios where you're moving an on-prem application to the cloud and the application writes data to either local or network attached disk.
You may also find this article useful: https://learn.microsoft.com/en-us/azure/storage/common/storage-decide-blobs-files-disks
UPDATE
Regarding uploading files from local computer to Azure Storage, there are actually many options available:
Use a Storage Explorer like Microsoft's Storage Explorer.
Use AzCopy command-line tool.
Use Azure PowerShell Cmdlets.
Use Azure CLI.
Write your own code using any available Storage Client libraries or directly consuming REST API.
As the title says, I'm looking for a way to access an azure files share (in preview) directly from an azure website. I cannot use any REST API or anything like this and I was looking for the possibility of mounting a SMB share directly into the website (through the new portal or any other way).
I found the following links, from which I understand that this is still under review (http://feedback.azure.com/forums/169385-web-apps-formerly-websites/suggestions/6084609-allow-map-azure-file-share-microsoft-azure-file-s) and also a SO question (Can the new Azure File Service be used from Azure WebSites?) that doesn't answer my question.
To be honest and for the sake of giving more details, my scenario is pretty simple - I have some websites and also some virtual machines that should access the files from the azure files service. Regarding the VM, the approach is pretty straight forward and easy but regarding the WebSites, I don't find any way at this moment.
On the other hand, regardless of the answer to the above question, does it make sense to (or do I have the possibility to) enable CDN over an Azure Files Share?
Thank you very much.
As of today, no single technology will serve your purpose. You can't use File Service as you don't have the capability to mount a share in an Azure Website as well as it is not suited for streaming purposes (all access to files there need to be authorized and there's no concept of Shared Access Signature in File Service today).
I guess, you would have to pick one of the two technologies (Blob Service and File Service) and make some compromises to make it work in both Websites and Virtual Machines.
Assuming you go with File Service, then you can mount them in the Virtual Machine and do the processing on the files there. On the website front, you would need to use Storage Client library to download the relevant files in some folder in your website and stream those files from there.
Assuming you go with Blob Service, then you can simply stream them in your website directly from blob storage (no need to have those files in your website). In the Virtual Machine, when you need to process those files (blobs), you would simply download them to your VM for processing and then re-upload them in blob storage.
Does it make sense to (or do I have the possibility to) enable CDN
over an Azure Files Share?
Currently it is not possible to serve Azure File Service files via CDN.
We have a solution which allows our customers to upload SCORM (http://scorm.com/scorm-explained/) packages which are basically mini websites, just html pages and a predefined javascript interface which allows the packages to talk to our site. This all works fine on a self hosted IIS machine as we build up a folder structure of each SCORM module within the website root and can allow the user to open each one and complete the course.
However these can be quite large, containing multiple videos, etc. and while on a self hosted machine we can place the site on a large hard drive how would this work if we wanted to migrate the solution to azure cloud services. I have read in several places the site must be stateless and the VM can be re-imaged at any time, does that mean that we shouldn't store anything in the folder structure that wasn't part of the original package? Is there a way to configure a shared permanent folder for our websites to use?
In a word BlobStorage!
As you quite rightly point out, Azure VMs are stateless so you need a single, persisted repository for your data / file, and blob storage easily fits the bill.
Without further information on precise requirements or how you explicitly need to reference these files, it's a bit hard to suggest the best way for you. However, here's a couple of things...
This video provides a quick overview of blob storage and retrieving items directly from BlobStorage. If you scenario requires you to serve up or store files on the local machine, this might be a good starting point.
BlobStorage supports direct access over HTTP and HTTPS, so you may want to simply reference files directly from BlobStorage (something that shouldn't be a problem as they're web assets). Check out the HTTP section in this article for the URL format. Note: you can also secure up these blob using SharedAccessSignatures if you want to restrict access.
There's no reason why you cannot use a standard VM then map a network drive using the Microsoft Azure File Service. I can't say I've personally done this, I used something else, but the concept a shared resource existing in BlobStorage is very doable.
HTH
I have a Azure website that allows customers to upload documents. There are a lot of documenet (~200Gb so far).
I need a way to backup the documents to another location (Azure or other location), or have live replication to a server. Is there anything I can use that will do this?
Have you looked at Azure storage redundancy options? The geo redundant option might solve your replication need.
http://blogs.msdn.com/b/windowsazurestorage/archive/2013/12/04/introducing-read-access-geo-replicated-storage-ra-grs-for-windows-azure-storage.aspx
You can use Cloudberry. Here is the link to their website. The tool will not give you cloud storage, but will assist in backup process. It can back up either to the cloud or to the local storages.
We are looking to replace our normal Win2008 R2 File server with a Azure Cloud based solution. Our Current use scenario is something like this:
Server A, Location A
Hosts 4gb of Data
Server B, Location B
Hosts 4gb of Data
Server A and B are Replicated.
Users access Data like:
\Domain-Name\DFSShare\ShareName
Many have Mapped Drives to the DFS Shares.
Can I use the Azure Storage to Replace this? I'd like to still have the users map Drives to the same shared storage areas on Azure.
I Would also need the same Security Permissions setup so all the users have the same permissions that they do for the Existing File Server storage. So I'd like Active Directly Integration.
Is this something that Azure Storage can do for me?
Thanks!
Might I recommend a product like StorSimple? This will accomplish the type of functionality you're after without need for costly development.
You can install a virtual server in Azure Iaas with a VNET and have a site-to-site VPN to your office. For users not in the office, you can have a VPN solution or install WebDAV on the file server and map drives over the internet.
You cannot direct-map local-server drive letters to cloud storage (which is triple-replicated), although individual Compute instances may do so. Also, mounting an NTFS volume inside a blob is a single-writer scenario, though multiple readers may snapshot and access these drives. All this to say no, this won't replace your network shares.
As #Brent pointed out, StorSimple is a good solution to explore, since it's an appliance that provides network share replication to blob storage transparently (and encrypts data prior to pushing it to Windows Azure). This would at least obviate the need for maintaining your replication server, as well as providing offsite replication.
I wrote a file system driver using the dokan library, which you can take a look at here:
https://github.com/richorama/AzureBlobDrive
I wouldn't say it was lightning fast, and it doesn't implement every feature of NTFS, but it might work for you.