I have four projects. One is a common project for other three projects.
Other three project build pipeline are depend on common build pipeline. When common build pipeline is in progress, other three build pipeline should be wait until common build complete. How to achive this in on premise azure devops?
How to achive this in on premise azure devops?
You could add a PowerShell task at the beginning of the other three pipelines.
Here is Powershell script sample:
$token = "PAT"
$url="https://{instance}/{collection}/{project}/_apis/build/definitions/{definitionId}?includeLatestBuilds=true&api-version=5.1"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method Get -ContentType application/json
$buildid = $response.latestBuild.id
$success = $false
do{
try{
$Buildurl2 = "https://{instance}/{collection}/{project}/_apis/build/builds/$($buildid)?api-version=5.0"
$Buildinfo2 = Invoke-RestMethod -Method Get -ContentType application/json -Uri $Buildurl2 -Headers #{Authorization=("Basic {0}" -f $token)}
$BuildStatus= $Buildinfo2.status
$result = $Buildinfo2.result
echo $result
echo $BuildStatus
if($BuildStatus -eq "completed") {
write-output "No Running Pipeline, starting Next Pipeline"
$success = $true
} else {
Write-output "Pipeline Build In Progress, Waiting for it to finish!"
Write-output "Next attempt in 30 seconds"
Start-sleep -Seconds 30
}
}
catch{
Write-output "catch - Next attempt in 30 seconds"
write-output "1"
Start-sleep -Seconds 30
# Put the start-sleep in the catch statemtnt so we
# don't sleep if the condition is true and waste time
}
$count++
}until($count -eq 2000 -or $success -eq $true )
if ($result -ne "succeeded")
{
echo "##vso[task.logissue type=error]Something went very wrong."
}
if(-not($success)){exit}
Explanation:
This powershell script runs the following two Rest APIs：
Definitions - Get
Builds - Get
The script checks the status of the pipeline(by polling) that is in process. If the pipeline is completed and the result is successful, it will run the other three pipelines. Or it will wait for the pipeline finishing the build.
Result Sample:
I am trying to work on an existing script that I had some assistance from in another thread. With some member assistance I was able to get my script to run using the "ThreadJob" module, however I was hoping I can also make use of runspacepools in conjunction with the ThreadJob to make it run faster.
In my code I post below, I am printing out a line to notify me that the function get's called. And I can see it is getting called. So it makes me think that line 56 and line 59 are incorrectly being called and I can't figure out how to call them.
if I run the "$rootPath\UpdateContacts\UpdateContacts.ps1" file manually through powershell ISE, it runs fine (obviously outside of the runspace), but I'd like to try and get it to work within the runspacepool.
Here is what I'm working with. I think I am pretty close.
begin
{
CLS
[switch]$MultiThread=$true
$rootPath = $(Split-path $MyInvocation.MyCommand.path -Parent)
$userEmail = "user#domain.com"
$SessionState = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1, 10,$Sessionstate, $Host)
$RunspacePool.Open()
$Jobs = #()
}
process
{
function ContactUpdater()
{
##################### Start Comparing Data #####################
#Our Array of values we will be comparing
[array]$CompareValues = "FirstName","MiddleName","LastName","DisplayName","Email","Mobile","TelephoneNumber","Title","Dept","Company"
for($i=0; $i -lt $CompareValues.Count; $i++)
{
#First let's create 2 variables that will hold the info we want
$A = ($Users).($CompareValues[$i])
$B = ($Contacts).($CompareValues[$i])
##################### Update Contacts #####################
#Only Run if there are contacts; otherwise there is nothing for us to compare
if(($NULL -ne $B))
{
#Displays all differences
#$Differences = [string[]]([Linq.Enumerable]::Except([object[]]$a, [object[]]$b) + [Linq.Enumerable]::Except([object[]]$b, [object[]]$a))
#Displays what accounts we need to import
$NeedsToBeAdded = [string[]]([Linq.Enumerable]::Except([object[]]$a, [object[]]$b))
#Displays what accounts we need to delete because they no longer exist
$NeedsToBeDeleted = [string[]]([Linq.Enumerable]::Except([object[]]$b, [object[]]$a))
}
}
##################### Import All Contacts #####################
if($NULL -eq $Contacts)
{
Write-Host "I am in the import"
# Load UpdateContacts function in memory
. "$rootPath\UpdateContacts\UpdateContacts\UpdateContacts.ps1"
#Write-host "Importing Contacts. This could take several minutes."
& "$rootPath\UpdateContacts\UpdateContacts.ps1"
}
}
if($MultiThread)
{
foreach($userEmail in $EmailAddress)
{
try
{
##################### Create Contact Folder #####################
if($NULL -eq $folderId)
{
$start = [datetime]::UtcNow
Write-Host "Creating Contacts Folder"
Try
{
while($NULL = (Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders/$folderId" -Headers $headers -Method get))
{
$NewContactFolder = Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders" -Body $ContactsFolderBody -Headers $headers -Method post -ContentType 'application/json'
sleep -Milliseconds 1
$folderId = $($NewContactFolder.id)
}
}
Catch
{
Out-Null
}
Add-Content "$rootPath\progress.txt" "`t`tCreated Contacts Folder in: $('{0:N2}' -f ([datetime]::UtcNow - $start).TotalSeconds) seconds"
Add-Content "$rootPath\progress.txt" ""
}
##################### Getting All User Info #####################
$start = [datetime]::UtcNow
$Users = & $rootPath\GetUserInfo\GetUserInfo.ps1
Add-Content "$rootPath\progress.txt" "`t`tFinished Getting all User Info in: $('{0:N2}' -f ([datetime]::UtcNow - $start).TotalSeconds) seconds"
Add-Content "$rootPath\progress.txt" ""
##################### Getting Contact Info #####################
if($NULL -ne $folderId)
{
$start = [datetime]::UtcNow
$Contacts = & $rootPath\GetContactInfo\GetContactInfo.ps1
Add-Content "$rootPath\progress.txt" "`t`tFinished Getting all Contact Info in: $('{0:N2}' -f ([datetime]::UtcNow - $start).TotalSeconds) seconds"
Add-Content "$rootPath\progress.txt" ""
}
##################### Import Contacts #####################
$start = [datetime]::UtcNow
CLS
if($NULL -eq $ImportMsg)
{
Write-host "Importing Contacts. This could take several minutes."
$ImportMsg = "Ran"
}
$ContactImporter = ContactUpdater
Add-Content "$rootPath\progress.txt" "`t`tFinished Importing Contact Info in: $('{0:N2}' -f ([datetime]::UtcNow - $start).TotalSeconds) seconds"
}
catch
{
$LogFile = "$rootPath\log.txt"
$errcond = $_.Exception.Message
$timestamp = (get-date).DateTime
"Time of exception: $timestamp" | Out-File $LogFile -Append
"User: $userEmail" | out-file $LogFile -Append
$errcond | out-file -FilePath $LogFile -append
}
1..10 | Foreach-Object {
$PowershellThread = [powershell]::Create()
$PowershellThread.RunspacePool = $RunspacePool
$PowershellThread.AddScript($ContactImporter).AddArgument($userEmail)
$Jobs += $PowershellThread.BeginInvoke()
}
}
}
}
end
{
if($MultiThread)
{
while ($Jobs.IsCompleted -contains $false)
{
Start-Sleep -Milliseconds 100
}
$RunspacePool.Close() | Out-Null
$RunspacePool.Dispose() | Out-Null
}
}
The part in the "Import all Contacts" section within the ContactUpdater() Function, should call the script:
& "$rootPath\UpdateContacts\UpdateContacts.ps1"
That script looks like this:
# Save the function in a scriptBlock, we need this
# so we can pass this function in the scope of the ThreadJobs
$updateContacts = "function UpdateContacts { $function:updateContacts }"
# Define the Number of Threads we are going to use
# (Get-CimInstance win32_processor).NumberOfLogicalProcessors
# Can give you a good perspective as to how many Threads is safe to use.
$numberOfThreads = 10
# $users is the array we want to process with
# the UpdateContacts function.
# Grouping the users in chunks so each running Job can process
# a chunk of users. Each chunk will contain around 50 users to process.
$groupSize = [math]::Ceiling($users.Count / $numberOfThreads)
$counter = [pscustomobject]#{ Value = 0 }
$chunks = $users | Group-Object -Property {
[math]::Floor($counter.Value++ / $groupSize)
}
foreach($chunk in $chunks)
{
# Capture this chunk of users in a variable
$thisGroup = $chunk.Group
# This is what we are running inside the scope
# of our threadJob
$scriptBlock = {
# Pass our variables to this scope
$UPN = $using:UPN
$folderID = $using:folderId
$headers = $using:headers
$contactsBody = $using:contactsBody
$ImportMsg = $using:ImportMsg
# First we need to define the function inside this scope
. ([scriptBlock]::Create($using:updateContacts))
# Loop through each user
foreach($user in $using:thisGroup)
{
UpdateContacts -User $user
}
}
# ThrottleLimit is the number of Jobs that can run at the same time.
# Be aware, a higher number of Jobs running does NOT mean that the
# task will perform faster. This always depends on your CPU & Memory.
# And, this case in particular, the number of requests your URI is able to handle
Start-ThreadJob -ScriptBlock $scriptBlock -ThrottleLimit $numberOfThreads
}
# Now we should have 10 Jobs running at the same time, each Job
# is processing a chunk of 50 users aprox. (500 users / 10)
# the output of all Jobs:
$result = Get-Job | Receive-Job -Wait
# Free up memory:
Get-Job | Remove-Job
That code above, starts a threadjob and launches another function in "$rootPath\UpdateContacts\UpdateContacts\UpdateContacts.ps1"
And that script looks like this:
Function UpdateContacts($User)
{
#FirstName, MiddleName, LastName, DisplayName, SamAccountName, Email, Mobile, TelephoneNumber, Title, Dept, Company, Photo, ExtensionAttribute2
$ContactsBody = #"
{
"givenName" : "$($User.FirstName)",
"middleName" : "$($User.MiddleName)",
"surname" : "$($User.LastName)",
"fileAs" : "$($User.LastName)",
"displayName" : "$($User.DisplayName)",
"jobTitle" : "$($User.Title)",
"companyName" : "$($User.Company)",
"department" : "$($User.Dept)",
"mobilePhone" : "$($User.Mobile)",
"homePhones" : ["$($User.TelephoneNumber)"],
"emailAddresses":
[
{
"address": "$($User.Email)",
"name": "$($User.DisplayName)"
}
]
}
"#
Try
{
Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders/$folderId/contacts" -Headers $headers -Body $ContactsBody -Method Post -ContentType 'application/json' | Out-Null
#After each user clear the info
$User = $NULL
}
Catch
{
if($error)
{
$User
$error
pause
}
$_.Exception.Message
Write-Host "--------------------------------------------------------------------------------------"
$_.Exception.ItemName
}
}
I am trying to fetch Recent file from sharepoint which is updated every month. Instead of manually downloading the file in local, I want to directly fetch from sharepoint and do analysis.
My sharepoint url looks likes this:
https://abc.sharepoint.com/_layouts/15/sharepoint.aspx
Where abc is my organisation name like apple, wipro so on.
I tried following this link:
Python - Download files from SharePoint site
## Sharepoint access
## pip install Office365-REST-Python-Client
from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.sharepoint.client_context import ClientContext
ctx_auth = AuthenticationContext("https://abc.sharepoint.com/_layouts/15/sharepoint.aspx")
ctx_auth.acquire_token_for_user('**#abc.com', '***')
I get the error as:
An error occurred while retrieving auth cookies from
https://abc.sharepoint.com/_layouts/15/sharepoint.aspx/_vti_bin/idcrl.svc
False
Secondly, someone has shared me the link to sharepoint and I have access to download it.
It looks like:
https://abc-my.sharepoint.com/personal/persons_name_abc_com/_layouts/15/onedrive.aspx?view=4
Inside this there is folder named Analysis Files. In Analysis Files folder there is excel file which
I have to download.
How can I do it using python 3?
SharePoint App-Only is the older, but still very relevant, model of setting up app-principals. This model works for both SharePoint Online and SharePoint 2013/2016 on-premises and is ideal to prepare your applications for migration from SharePoint on-premises to SharePoint Online.
I am not familiar with Python3 but you can use the below code sample in your code as it uses REST API.
Please refer the following articles and let me know if you have any questions
SharePoint App-Only - https://sprider.blog/spol-download-file-from-library-using-sharepoint-app-only-access-and-powershell
Graph API - https://sprider.blog/download-sharepoint-file-using-microsoft-graph-api
How to get Tenant ID - https://sprider.blog/get-office-365-tenant-id-from-domain-name
Tenant Name - abc as per your code example.
Client Id and Secret can be retrived from SharePoint or Azure AD App Registration
###### Global Static Variables - Start ######
$grantType = "client_credentials"
$principal = "00000003-0000-0ff1-ce00-000000000000"
###### Global Static Variables - End ######
###### Global Tenant Specific Variables - Start ######
$m365TenantId = "yourtenantguid"
$targetHost = "yourtenantname.sharepoint.com"
$appClientId = "clientid-from-previous-section"
$appClientSecret = "clientsecret-from-previous-section"
###### Global Tenant Specific Variables - End ######
###### Site/File Path Variables - Start ######
$targetFolder = $PSScriptRoot
$siteRelativeUrl = "sites/yoursite"
$folderRelativeUrl = "your-document-library-name"
$fileName = "your-file-name.png"
###### Site/File Path Variables - Start ######
###### Helper Functions - Start ######
function Add-Working-Directory([string]$workingDir, [string]$logDir) {
if (!(Test-Path -Path $workingDir)) {
try {
$suppressOutput = New-Item -ItemType Directory -Path $workingDir -Force -ErrorAction Stop
$msg = "SUCCESS: Folder '$($workingDir)' for CSV files has been created."
Write-Host -ForegroundColor Green $msg
}
catch {
$msg = "ERROR: Failed to create '$($workingDir)'. Script will abort."
Write-Host -ForegroundColor Red $msg
Exit
}
}
if (!(Test-Path -Path $logDir)) {
try {
$suppressOutput = New-Item -ItemType Directory -Path $logDir -Force -ErrorAction Stop
$msg = "SUCCESS: Folder '$($logDir)' for log files has been created."
Write-Host -ForegroundColor Green $msg
}
catch {
$msg = "ERROR: Failed to create log directory '$($logDir)'. Script will abort."
Write-Host -ForegroundColor Red $msg
Exit
}
}
}
function Add-Log([string]$message, [string]$logFile) {
$lineItem = "[$(Get-Date -Format "dd-MMM-yyyy HH:mm:ss") | PID:$($pid) | $($env:username) ] " + $message
Add-Content -Path $logFile $lineItem
}
function Get-AccessToken {
try {
$message = "Getting Accesstoken..."
Add-Log $message $Global:logFile
$tokenEndPoint = "https://accounts.accesscontrol.windows.net/$m365TenantId/tokens/oauth/2"
$client_Id = "$appClientId#$m365TenantId"
$resource = "$principal/$targetHost#$m365TenantId"
$requestHeaders = #{
"Content-Type" = "application/x-www-form-urlencoded"
}
$requestBody = #{
client_id = $client_Id
client_secret = $appClientSecret
grant_type = $grantType
resource = $resource
}
$response = Invoke-RestMethod -Method 'Post' -Uri $tokenEndPoint -Headers $requestHeaders -Body $requestBody
$accesstoken = $response.access_token
$message = "Accesstoken received."
Add-Log $message $Global:logFile
return $accesstoken
}
catch {
$statusCode = $_.Exception.Response.StatusCode.value__
$statusDescription = $_.Exception.Response.StatusDescription
$message = "StatusCode: $statusCode"
Add-Log $message $Global:logFile
$message = "StatusDescription : $statusDescription"
Add-Log $message $Global:logFile
return $null
}
}
function Download-File([string]$fileUrl, [string]$targetFilePath) {
$accessToken = Get-AccessToken
if (![string]::IsNullOrEmpty($accessToken)) {
try {
$fileUri = New-Object System.Uri($fileUrl)
$wc = New-Object System.Net.WebClient
$wc.Headers.Add("Authorization", "Bearer $accessToken")
$job = $wc.DownloadFileTaskAsync($fileUri, $targetFilePath)
$message = "Downloading file $fileUrl at $targetFilePath."
Add-Log $message $Global:logFile
while (!$job.IsCompleted) {
sleep 1
}
if ($job.Status -ne "RanToCompletion") {
$message = "Failed to download file."
Add-Log $message $Global:logFile
}
else {
$message = "File downloaded."
Add-Log $message $Global:logFile
}
}
catch {
$statusCode = $_.Exception.Response.StatusCode.value__
$statusDescription = $_.Exception.Response.StatusDescription
$message = "StatusCode: $statusCode"
Add-Log $message $Global:logFile
$message = "StatusDescription : $statusDescription"
Add-Log $message $Global:logFile
$message = "Failed to download file."
Add-Log $message $Global:logFile
}
}
else {
$message = "Unable to get Accesstoken."
Add-Log $message $Global:logFile
}
}
###### Helper Functions - End ######
###### Main Program - Start ######
###### Log Setup - Start ######
$currentDirectoryPath = Split-Path -Parent $MyInvocation.MyCommand.Definition
$workingDirectory = $currentDirectoryPath
$logDirectoryName = "Logs"
$logDirectory = "$workingDirectory/$logDirectoryName"
$logFileName = "$(Get-Date -Format "yyyyMMddTHHmmss")_downloadjobexecution.log"
$Global:logFile = "$logDirectory/$logFileName"
Add-Working-Directory $workingDirectory $logDirectory
###### Log Setup - Start ######
Write-Host -ForegroundColor Yellow "WARNING: Minimal output will appear on the screen."
Write-Host -ForegroundColor Yellow " Please look at the log file '$($logFile)'"
$message = "**************************************** SCRIPT STARTED ****************************************"
Add-Log $message $Global:logFile
###### Download File - Start ######
$targetFilePath = Join-Path $targetFolder $fileName
$fileUrl = "https://$targetHost/$siteRelativeUrl/_api/Web/GetFolderByServerRelativeUrl('$folderRelativeUrl')/Files('$fileName')/`$value"
Download-File $fileUrl $targetFilePath
###### Download File - End ######
$message = "**************************************** SCRIPT COMPLETED ****************************************"
Add-Log $message $Global:logFile
###### Main Program - End ######
The error shows that there is something wrong when you get authorization. You could connect to SharePoint site like below:
tenant_url= "https://{tenant}.sharepoint.com"
ctx_auth = AuthenticationContext(tenant_url)
site_url="https://{tenant}.sharepoint.com/sites/{yoursite}"
ctx_auth.acquire_token_for_user("username","password"):
trying sending some mail using function mail()
$passage_ligne = "\r\n";
$boundary = "-----=".md5(rand());
$headers = 'From:'.$nom.' <'.$email.'>' . "\r\n";
$headers .= 'Return-Path: '.$nom.' <'.$email.'>'.$passage_ligne;
$headers .= "Message-ID:<".time()." TheSystem#".$_SERVER['SERVER_NAME'].">".$passage_ligne;
$headers .= "X-Mailer: PHP v".phpversion().$passage_ligne;
$headers.= "MIME-Version: 1.0".$passage_ligne;
$headers.= "Content-Type: multipart/related; boundary=\"$boundary\"".$passage_ligne;
if(file_exists($file))
{
$file_type = filetype($file);
$file_size = filesize($file);
$handle = fopen($file, 'r') or die('File '.$file.'can t be open');
$content = fread($handle, $file_size);
$content = chunk_split(base64_encode($content));
$f = fclose($handle);
$corps = '--'.$boundary."\r\n";
$corps .= 'Content-type:'.$file_type.';name='.$file."\r\n";
$corps .= 'Content-transfer-encoding:base64'."\r\n";
$corps .= "Content-Disposition: attachment; filename='".$file."'\r\n";
$corps .= $content."\r\n";
}
In the undelivered message, it says:
Host or domain name not found. Name service error for name=xxx-xxx.com type=A: Host not found
Can't see whats wrong please?
trying to send email via function mail()
I receive an email without files. It was working sending only one file too.
But this doesn't work sending multiple files.
I don't understand how to use boundary? So my code doesn't work.
$passage_ligne = "\r\n";
$boundary = "-----=".md5(rand());
$boundary_alt = "-----=".md5(rand());
$headers = 'From:'.$nom.' <'.$email.'>' . "\r\n";
$headers .= 'Return-Path: '.$nom.' <'.$email.'>'.$passage_ligne;
$headers .= "Message-ID:<".time()." TheSystem#".$_SERVER['SERVER_NAME'].">".$passage_ligne;
$headers .= "X-Mailer: PHP v".phpversion().$passage_ligne;
$headers.= "MIME-Version: 1.0".$passage_ligne;
$headers.= "Content-Type: multipart/mixed;".$passage_ligne." boundary=\"$boundary\"".$passage_ligne;
$corps = $passage_ligne."--".$boundary.$passage_ligne;
$corps.= "Content-Type: multipart/alternative;".$passage_ligne." boundary=\"$boundary_alt\"".$passage_ligne;
$corps.= $passage_ligne."--".$boundary_alt.$passage_ligne;
$corps.= "Content-Type: text/plain; charset=\"ISO-8859-1\"".$passage_ligne;
$corps.= "Content-Transfer-Encoding: 8bit".$passage_ligne;
$corps.= $passage_ligne.$message.$passage_ligne;
$corps.= $passage_ligne."--".$boundary_alt.$passage_ligne;
if(isset($_FILES["file"]) && $_FILES['file']['name'] != "")
{
$nom_fichier = $_FILES['file']['name'];
$source = $_FILES['file']['tmp_name'];
$type_fichier = $_FILES['file']['type'];
$taille_fichier = $_FILES['file']['size'];
foreach($_FILES as $photo){
$tmp_name = $photo['tmp_name'];
$type = $photo['type'];
$name = $photo['name'];
$size = $photo['size'];
if (file_exists($tmp_name)){
if(is_uploaded_file($tmp_name)){
$file = fopen($tmp_name,'r');
$data = fread($file,filesize($tmp_name));
fclose($file);
$data = chunk_split(base64_encode($data));
}
$corps .= $passage_ligne."--".$boundary."--".$passage_ligne;
$corps .= "Content-Type: ".$type."; name=\"".$name."\"".$passage_ligne;
$corps .= "Content-Transfer-Encoding: base64".$passage_ligne;
$corps .= "Content-Disposition: attachment; filename='".$name."'".$passage_ligne;
$corps .= $passage_ligne.$data.$passage_ligne.$passage_ligne;
}
}
$corps.= $passage_ligne."--".$boundary."--".$passage_ligne;
Someone could explain this boundary to me please? How to use it?
Thanks.