downloading a file from SharePoint Online with PowerShell - sharepoint

Question

I have a requirement to download files from a sharepoint online document library using powershell
I've managed to get to the point where the download should happen but no luck.
I know its something to do with how I am using the stream/writer
any hints would be greatly appreciated
*Edit
No error messages are thrown just 0 length files in my local Directory
$SPClient = [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client")
$SPRuntime = [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime")
$webUrl = Read-Host -Prompt "HTTPS URL for your SP Online 2013 site"
$username = Read-Host -Prompt "Email address for logging into that site"
$password = Read-Host -Prompt "Password for $username" -AsSecureString
$folder = "PoSHTest"
$destination = "C:\\test"
$ctx = New-Object Microsoft.SharePoint.Client.ClientContext($webUrl)
$ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)
$web = $ctx.Web
$lists = $web.Lists.GetByTitle($folder)
$query = [Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery(10000)
$result = $lists.GetItems($query)
$ctx.Load($Lists)
$ctx.Load($result)
$ctx.ExecuteQuery()
#Edited the foreach as per #JNK
foreach ($File in $result) {
Write-host "Url: $($File["FileRef"]), title: $($File["FileLeafRef"]) "
$binary = [Microsoft.SharePoint.Client.File]::OpenBinaryDirect($ctx,$File["FileRef"])
$Action = [System.IO.FileMode]::Create
$new = $destination + "\\" + $File["FileLeafRef"]
$stream = New-Object System.IO.FileStream $new, $Action
$writer = New-Object System.IO.BinaryWriter($stream)
$writer.write($binary)
$writer.Close()
}

Answer 1

You could also utilize WebClient.DownloadFile Method by providing SharePoint Online credentials to download the resource from SharePoint Online as demonstrated below.
Prerequisites
SharePoint Online Client Components SDK have to be installed on the machine running the script.
How to download a file in SharePoint Online/O365 in PowerShell
Download-File.ps1 function:
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client")
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime")
Function Download-File([string]$UserName, [string]$Password,[string]$FileUrl,[string]$DownloadPath)
{
if([string]::IsNullOrEmpty($Password)) {
$SecurePassword = Read-Host -Prompt "Enter the password" -AsSecureString
}
else {
$SecurePassword = $Password | ConvertTo-SecureString -AsPlainText -Force
}
$fileName = [System.IO.Path]::GetFileName($FileUrl)
$downloadFilePath = [System.IO.Path]::Combine($DownloadPath,$fileName)
$client = New-Object System.Net.WebClient
$client.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $SecurePassword)
$client.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f")
$client.DownloadFile($FileUrl, $downloadFilePath)
$client.Dispose()
}
Usage
Download-File -UserName "username#contoso.onmicrosoft.com" -Password "passowrd" -FileUrl https://consoto.sharepoint.com/Shared Documents/SharePoint User Guide.docx -DownloadPath "c:\downloads"

Answer 2

I was able to download the file successfully with the following relevant code snippet. You should be able to extend it for your situation.
Add-Type –Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type –Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
$siteUrl = Read-Host -Prompt "Enter web URL"
$username = Read-Host -Prompt "Enter your username"
$password = Read-Host -Prompt "Enter password" -AsSecureString
$source = "/filepath/sourcefilename.dat" #server relative URL here
$target = "C:/detinationfilename.dat" #URI of the file locally stored
$ctx = New-Object Microsoft.SharePoint.Client.ClientContext($siteUrl)
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)
$ctx.Credentials = $credentials
[Microsoft.SharePoint.Client.FileInformation] $fileInfo = [Microsoft.SharePoint.Client.File]::OpenBinaryDirect($ctx,$source);
[System.IO.FileStream] $writeStream = [System.IO.File]::Open($target,[System.IO.FileMode]::Create);
$fileInfo.Stream.CopyTo($writeStream);
$writeStream.Close();

Answer 3

While the CSOM code above likely can be made to work I find it easier to use the web client method.
(from http://soerennielsen.wordpress.com/2013/08/25/use-csom-from-powershell/)
I've used the code below, to retrieve a bunch of files (metadata from CSOM queries) to a folder (using your $result collection, other params should be adjusted a bit):
#$siteUrlString site collection url
#$outPath path to export directory
$siteUri = [Uri]$siteUrlString
$client = new-object System.Net.WebClient
$client.UseDefaultCredentials=$true
if ( -not (Test-Path $outPath) ) {
New-Item $outPath -Type Directory | Out-Null
}
$result |% {
$url = new-object Uri($siteUri, $_["FileRef"])
$fileName = $_["FileLeafRef"]
$outFile = Join-Path $outPath $fileName
Write-Host "Downloading $url to $outFile"
try{
$client.DownloadFile( $url, $outFile )
}
catch{
#one simple retry...
try{
$client.DownloadFile( $url, $outFile )
}
catch{
write-error "Failed to download $url, $_"
}
}
}
The trick here is the
$client.UseDefaultCredentials=$true
which will authenticate the webclient for you (as the current user).

Answer 4

The direct and almost shortest answer to the question is simply:
$url = 'https://the.server/path/to/the/file.txt'
$outfile = "$env:userprofile\file.txt"
Invoke-WebRequest -Uri $url -OutFile $outfile -Credential (Get-Credential)
This works at least in Powershell 5.1...

Answer 5

So I gave up on this. it turned out to be much easier to write an SSIS script component to do the job.
I have awarded Soeren as he posted some code that will work for regular websites but not sodding SharePoint Online.
Thanks Sorean!

Answer 6

Short an easy approach to download a file from sharepoint online, using just powershell and sharepoint online url ( no pnp powershell )
This approach can also be used to perform Sharepoint REST queries, with just powershell and sharepoint REST api
# required MS dependencies
# feel free to download them from here https://www.microsoft.com/en-us/download/details.aspx?id=42038
Add-Type -Path 'C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll' -ErrorAction Stop
Add-Type -Path 'C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll' -ErrorAction Stop
# prepare passwords
$spCredential = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($user, $(ConvertTo-SecureString -AsPlainText $pass -Force))
# prepare and perform rest api query
$Context = New-Object Microsoft.SharePoint.Client.ClientContext($targetSiteUrl)
$Context.Credentials = $spCredential
try {
#this may return an error, but still will finish context setup
$Context.ExecuteQuery()
}
catch {
write-host "TODO: fix executeQuery() err 400 bug" -ForegroundColor Yellow
}
$AuthenticationCookie = $Context.Credentials.GetAuthenticationCookie($targetSiteUrl, $true)
$WebSession = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$WebSession.Credentials = $Context.Credentials
$WebSession.Cookies.SetCookies($targetSiteUrl, $AuthenticationCookie)
$WebSession.Headers.Add("Accept", "application/json;odata=verbose")
Invoke-WebRequest -Uri $spFileUrl -OutFile $outputFilePath -WebSession $WebSession -errorAction Stop
Where
$outputFilePath is the target output file in which you want to save the remote file.
$targetSiteUrl is the target sp site url.
$spFileUrl is the "[sharepoint file full url]"
$user plain text sp user email
$pass plain text sp user pass