How to logs (stdout / stderr) from all container pods azure Kubernetes to the event hub.
I can able to see all logs by Log Analytics workspaces >> Logs using an Azure query language.
I want to send all logs to the event hub.
Can anyone suggest on this?
You can easily forward container logs to Event Hubs via Fluent-Bit's Kafka output.
Here is Fluent-Bit documentation for Kafka - https://docs.fluentbit.io/manual/pipeline/outputs/kafka
And here is Kafka client integration with Event Hubs - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
Worked for me using fluentbit kafka output to Azure EventHub
td-agent-bit.conf
[INPUT]
Name tail
Path xxx.log
Refresh_Interval 10
[OUTPUT]
Name kafka
Match *
brokers xxx.xxx.windows.net:9093
topics xxx
rdkafka.security.protocol SASL_SSL
rdkafka.sasl.username $ConnectionString
rdkafka.sasl.password Endpoint=sb://xxx.xxx.windows.net/;SharedAccessKeyName=xxx;SharedAccessKey=xxx
rdkafka.sasl.mechanism PLAIN
[OUTPUT]
name stdout
match *
Inside docker container (MUST HAVE or broker down/ssl fail)
docker-compose.yml
version: "3.7"
services:
fluent-bit:
image: fluent/fluent-bit:1.6.2
container_name: fluentbit
restart: always
volumes:
- ./td-agent-bit.conf:/fluent-bit/etc/fluent-bit.conf
- ./xxx.log:/fluent-bit/etc/xxx.log:ro
Related
Problem:
Flink task manager reports: apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
Deployment overview:
A Java project to try out Stateful Functions.The streaming app reads messages from Kafka, processes messages and sends the final result to kafka egress.
Deployed on Azure:
Azure Event Hub (Kafka Endpoint) as ingress and egress
Azure Kubernetes Service as k8s deployment
Azure Data Lake Gen 2 as storage for checkpoint
Deployment is good, job manager and task manager has been launched, then I see task failed to run due to the exception
Diagnostics:
I created a simple Java consumer with the identical kafka config,
just with a different consumer group. The Java app works well both
on my laptop and in AKS (deployed in the same namespace as the
stateful function app is) So I get a conclusion that the Event Hub
and my kafka config are both good.
I checked the task manager log (kubectl logs xxx), and the kafka properties have been correctly loaded. The sasl.jaas.config shows as "sasl.jaas.config = [hidden]" but I assume this is by design.
My Kafka Settings:
I'm using the following config:
kind: io.statefun.kafka.v1/ingress
spec:
id: io.streaming/eventhub-ingress
address: xxxx.servicebus.windows.net:9093
consumerGroupId: group-receiver-00
startupPosition:
type: group-offsets
topics:
- topic: streaming-topic-rec-32
valueType: streaming.types/rec
targets:
- streaming.fns/bronze_rec
- topic: streaming-topic-eng-32
valueType: streaming.types/eng
targets:
- streaming.fns/bronze_eng
properties:
- request.timeout.ms: 60000
- security.protocol: SASL_SSL
- sasl.mechanism: PLAIN
- sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="primary connection string of the event hub ns";
Can anyone help me with this? Thank you!
Resolved after reducing replicas of task manager. No config changed
I'm using the Python 3.8 SDK for Azure service bus, (azure-servicebus v. 0.50.3). I use the following code to send a message to a topic ...
service = ServiceBusService(service_namespace,
shared_access_key_name=key_name,
shared_access_key_value=key_value)
msg = Message(json.dumps({'type': 'my_message'}))
service.send_topic_message(topic_name, msg)
How do I create a Docker image that runs the service bus with a topic or two already created? I found this image
version: '3.7'
services:
azure_sb:
container_name: azure_sb
image: microsoft/azure-storage-emulator
tty: true
restart: always
ports:
- "10000:10000"
- "10001:10001"
- "10002:10002"
but I'm unclear how to connect to it using the code I have or if the above is even a valid service bus image.
Azure Service Bus does not provide a docker image. The image that you are using (microsoft/azure-storage-emulator) is for the Azure Storage system, which can provide similar queuing capabilities with Azure Storage Queues. For more details check out How to use Azure Queue storage from Python.
If you need to use Azure Service Bus locally, check out the GitHub Issue: Local Development story?. TLDR: Use AMQP libraries and connect to another AMQP provider for local, and swap out for Service Bus in production.
There is an AKS cluster in Azure with a microservices app (NodeJS) deployed.
The app is streaming logs to Application Insights (and to Log Analytics) using application insights.
All settings are by default: maxBatchSize & maxBatchIntervalMs.
When I check the pod logs using kubectl logs {POD_NAME} -n {NAMESPACE_NAME} I see the following output:
---
2020-05-05T00:22:24.851Z info xxxxxxxxx-a94b-4666-9e83-31e945e1ee15 0oaXXXXXXXXXX PUT /storages/1XXXXXX007475 xxxMiddleware
xxxxMiddleware ended
Postgres pool raised an error. Error: read ETIMEDOUT
---
In Log Analytics using Kusto query I see the same:
LogEntrySource stderr
LogEntry Postgres pool raised an error. Error: read ETIMEDOUT
TimeGenerated 2020-05-05T00:58:09
Computer aks-agentpool-xxxxx-3
ContainerID 4a5e5e69957917578d8b18b59628ec3f21cd1dd19b5d66ca1xxxxxxxxxx
I'm wondering why the difference in timestamps is ~ 35+ minutes, i.e. 2020-05-05T00:58:09 - 2020-05-05T00:22:24.851Z
Yes, the difference is about 30 min. I guess this could be the difference in the timezone. kubectl reports logs in the timezone of the pod where as LogAnalytics might be reports it into UTC timezone or in your browser timezone. Please check the timezones of pod and the browser and see if this makes sense.
Hopefully this should solve the problem.
I am trying to use spark on Kubernetes. Idea is to using spark-submit to k8s cluster which is running prometheus operator. Now I know that prometheus operator can respond to ServiceMonitor yaml but I am confused how to provide some of the things required in the YAML using spark-submit
Here is the YAML:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: sparkloads-metrics
namespace: runspark
spec:
selector:
matchLabels:
app: runspark
namespaceSelector:
matchNames:
- runspark
endpoints:
- port: 8192 ---> How to provide the name to port using `spark-submit`
interval: 30s
scheme: http
You cannot provide additional ports and their names to the Service created by SparkSubmit yet (Spark v2.4.4). Things can change in the later versions.
What you can do is create additional Kubernetes Service (Spark Monitoring Service, eg. of type Cluster IP) per Spark job after the Job submission with SparkSubmit, for instance running spark-submit ... && kubectl apply ... . Or use any of the available Kubernetes clients with the language of your choice.
Note that you can use Kubernetes OwnerReference to configure automatic Service deletion/GC on Spark Driver Pod deletion.
Then you can supply the ServiceMonitor's via the Prometheus Operator Helm values:
prometheus:
additionalServiceMonitors:
- name: spark-metrics # <- Spark Monitoring Service name
selector:
matchLabels:
k8s-app: spark-metrics # <- Spark Monitoring Service label
namespaceSelector:
any: true
endpoints:
- interval: 10s
port: metrics # <- Spark Monitoring Service port name
Be aware of the fact that Spark doesn't provide a way to customize Spark Pods yet, so your Pod ports which should expose metrics are not exposed on a Pod level and won't be accessible via Service. To overcome it you can add additional EXPOSE ... 8088 statement in the Dockerfile and rebuild Spark image.
This guide should help you to setup Spark monitoring with PULL strategy using for example Jmx Exporter.
There is an alternative (though it is recommended only for short-running Spark jobs, but you can try it in your environment if you do not run huge workloads):
Deploy Prometheus Pushgateway and integrate it with your Prometheus Operator
Configure Spark Prometheus Sink
By doing that your Spark Pods will PUSH metrics to the Gateway and Prometheus will PULL them from the Gateway in order.
You can refer the Spark Monitoring Helm chart example with the Prometheus Operator and Prometheus Pushgateway combined.
Hope it helps.
I would like to configure continuous integration from VSTS to Azure Container Registry and then to WebApp.
Here's my docker-compose.yml file: As you can see I'm using an Asp.Net core + mssql.
version: '3'
services:
api:
image: tbacr.azurecr.io/myservice/api
container_name: api
build:
context: ./Api
dockerfile: Dockerfile
ports:
- "8000:80"
depends_on:
- db
db:
image: "microsoft/mssql-server-linux"
container_name: mssql
environment:
SA_PASSWORD: "testtest3030!"
ACCEPT_EULA: "Y"
MSSQL_PID: "Developer"
ports:
- "127.0.0.1:8001:1433"
I have pushed my image using VSTS to Azure Container Registry.
Here's my Azure Container Registry:
But when I try to open URI in browser - I get an error message:
Service Unavailable which is HTPP 503 code.
Should I configure something more?
I think you can only deploy one container to an App Service. You have two images in your docker-compose file. You might consider using either the "Azure SQL Database" or the "Azure Database for MySQL" if you need a database.
You can set up continuous deployment of a custom image from your Azure Container Registry by right clicking the tag of an image in the repository and choose "deploy to web app". Every time you update the image also the web app will be updated.
If you want to deploy Microservices use Service Fabric, Azure Container Service (ACS) or Azure Kubernetes Service (AKS).
PS: You have uploaded the image of the App Service, not of the Container Registry.