Hello everyone, I’m experiencing a recurring issue with high CPU usage on my server when uploading larger files through a Docker container while using aaPanel. I hope to open an interesting discussion and get some insights from the community and aaPanel technicians. Here’s a breakdown of the problem: System and Setup Server OS: CentOS Stream 8 x86_64 Control Panel: aaPanel 7.16.0 (Free) Docker Environment: Running several containers, including open-webui, watchtower, and pipelines. if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Issue Details High CPU Usage During File Uploads: When uploading files over 5 MB to open-webui, my Docker container’s CPU usage spikes dramatically, sometimes reaching over 600%. This affects the server's overall performance. Smaller files (around 2 MB or less) upload without significant CPU increase. Resource Consumption: The open-webui Docker container consumes around 73% of the CPU during regular operations, and this goes up when handling larger files. Memory usage remains stable and doesn’t seem to be the limiting factor. Environment Comparison: I’m running the same project in another environment without aaPanel, and it works smoothly, even with large file uploads. This leads me to believe that aaPanel might be interacting with Docker in a way that impacts resource allocation or process management during intensive tasks. if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Steps I've Tried So Far Increased File Upload Limits: Adjusted the file upload size limit in Docker and open-webui, but it didn’t resolve the CPU spike. Resource Limiting: Tried setting CPU limits on the open-webui Docker container, but the usage remains high when handling larger files. aaPanel Configurations: Checked aaPanel settings to see if there were any conflicts with Docker’s resource management, but no clear issue was found. if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Request for Assistance I'm looking for guidance or suggestions from those with experience in managing Docker containers within aaPanel. Specifically: Has anyone else experienced high resource usage with aaPanel while running Docker containers? Are there any recommended configurations in aaPanel for better compatibility with Docker? Any insights on optimizing file uploads in Docker to prevent CPU spikes? if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } I appreciate any help or suggestions! Let’s make this an engaging thread for anyone else facing similar issues. Thank you!

Hello, thank you for your continued support and willingness to investigate this further! To facilitate testing, I can provide access to the Docker container in two possible ways. Please let me know which option is more convenient for you: Option 1: Provide Access to a Docker Image Repository I can upload the Docker image of open-webui to a public repository like Docker Hub or a private repository if you prefer. Once uploaded, I can share the repository link with you for easy download and testing on your end. This option allows you to deploy and configure the container independently. if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Option 2: Provide Temporary Access to My Server Alternatively, I can provide you with temporary access to my server where aaPanel and the open-webui container are already configured. This option would allow you to observe the issue directly in my setup, including all configurations and resource usage. If this approach is preferred, please advise on any specific access requirements or security protocols you’d like me to follow. if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } I’m happy to proceed with whichever method works best for you. Please let me know, and I’ll set it up accordingly! Thank you!

High CPU Usage on Docker Container with aaPanel During Large File Uploads

PTYNETWORK

Hello everyone,

I’m experiencing a recurring issue with high CPU usage on my server when uploading larger files through a Docker container while using aaPanel. I hope to open an interesting discussion and get some insights from the community and aaPanel technicians. Here’s a breakdown of the problem:
System and Setup

Server OS: CentOS Stream 8 x86_64
Control Panel: aaPanel 7.16.0 (Free)
Docker Environment: Running several containers, including open-webui, watchtower, and pipelines.

Issue Details

High CPU Usage During File Uploads:
    When uploading files over 5 MB to open-webui, my Docker container’s CPU usage spikes dramatically, sometimes reaching over 600%. This affects the server's overall performance.
    Smaller files (around 2 MB or less) upload without significant CPU increase.

Resource Consumption:
    The open-webui Docker container consumes around 73% of the CPU during regular operations, and this goes up when handling larger files.
    Memory usage remains stable and doesn’t seem to be the limiting factor.

Environment Comparison:
    I’m running the same project in another environment without aaPanel, and it works smoothly, even with large file uploads.
    This leads me to believe that aaPanel might be interacting with Docker in a way that impacts resource allocation or process management during intensive tasks.

Steps I've Tried So Far

Increased File Upload Limits: Adjusted the file upload size limit in Docker and open-webui, but it didn’t resolve the CPU spike.
Resource Limiting: Tried setting CPU limits on the open-webui Docker container, but the usage remains high when handling larger files.
aaPanel Configurations: Checked aaPanel settings to see if there were any conflicts with Docker’s resource management, but no clear issue was found.

Request for Assistance

I'm looking for guidance or suggestions from those with experience in managing Docker containers within aaPanel. Specifically:

Has anyone else experienced high resource usage with aaPanel while running Docker containers?
Are there any recommended configurations in aaPanel for better compatibility with Docker?
Any insights on optimizing file uploads in Docker to prevent CPU spikes?

I appreciate any help or suggestions! Let’s make this an engaging thread for anyone else facing similar issues.

Thank you!

aaPanel_Kern

Hello, is there a mirror name that we can download? How do we test this?
aapanel and docker interact using API. Generally there will be no relevant impact
Can you check which process is using high CPU usage?

PTYNETWORK

Hello, thank you for your response!

I appreciate your suggestion, and I’m happy to provide more details to help diagnose the issue.
Additional Information

Project Access:
    The project is deployed on a Docker container called open-webui. Unfortunately, there’s no public mirror, but I can share a test scenario to replicate the issue if needed.
    Alternatively, if there's a way to securely share the container setup or environment specs, please let me know.

High CPU Usage Details:
    I checked the CPU usage in more detail, and here’s what I observed:
        The open-webui container itself spikes in CPU usage, often reaching around 600% during large file uploads (files over 5 MB).
        Other containers, like watchtower and pipelines, remain idle and don’t seem to impact performance.

Process Monitoring:
    Using docker stats and htop, I confirmed that the open-webui container’s process specifically consumes the bulk of the CPU. When uploading a file over 5 MB, it triggers a sudden rise in CPU activity.
    The rest of the system processes remain stable, which points to a potential bottleneck within the Docker container or how the open-webui app processes larger files.

aaPanel API Interaction:
    I understand that aaPanel interacts with Docker via API, so it shouldn’t impact performance directly. However, I suspect that aaPanel’s resource management could be influencing Docker’s CPU allocation. Have there been any known issues with high resource consumption in similar setups?

Request for Further Guidance

Could there be any specific settings in aaPanel to optimize Docker container performance, particularly for applications that handle large file uploads?
Would you recommend testing any specific configurations to adjust how Docker or aaPanel manages resource allocation?

Thank you for your support. I’m open to trying any recommendations or sharing more details if it helps!

aaPanel_Kern

Hello, it is recommended that you stop aapanel before testing. Currently, there is no special optimization for aapanel and docker. The default configuration of docker is used.
It may be caused by a problem with the open-webui container itself. It is recommended that you check the specific process.

PTYNETWORK

Response to aaPanel Admin: Further Investigation and Observations

Hello, thank you for your continued support.

I've done some additional testing to further isolate the issue. Here’s what I’ve observed and the steps I've taken:
Testing Across Different Environments

Comparison Environment:
    I am running the exact same Docker image (openwebui:cuda and openwebui:default) on another server environment with Ubuntu 23.03 and Ollama 3.14.
    In that setup, I don’t experience any issues with large file uploads, even when uploading files over 5 MB.
    This leads me to believe the issue may not lie within the open-webui container itself but could be influenced by the aaPanel setup.

aaPanel Docker Configuration:
    I initially deployed openwebui:cuda on aaPanel version 7.11 using the Docker app within aaPanel. I have since upgraded to aaPanel 7.16, which includes an updated Docker application interface that simplifies deployment.
    The issue has persisted through both versions of aaPanel.

Proxy Configuration and SSL:
    To make the Docker container accessible externally, I set up a reverse proxy on aaPanel, redirecting a subdomain to the local IP of the Docker container (open-webui).
    SSL is enabled for this subdomain. However, the Docker container itself is accessible only over HTTP, not HTTPS.
    I’m wondering if there could be a conflict between the SSL settings on the subdomain and the HTTP-only nature of the Docker container, though it’s unclear if this would contribute to CPU usage.

Next Steps

Further Testing on a Fresh Setup:
    To rule out any environmental issues specific to my current setup, I’m planning to deploy aaPanel on a new VPS with Ubuntu Server 24.04 and then reinstall the open-webui Docker container there.
    This should help me determine whether the problem is linked to my current server’s configuration or aaPanel’s interaction with Docker.

Request for Insights

Are there any known limitations or conflicts between aaPanel's reverse proxy and Docker containers that could lead to high CPU usage?
Is there a recommended method for using SSL with Docker containers in aaPanel to avoid potential conflicts with HTTP-only configurations?

Thank you again for your guidance. I’ll update this thread with my findings from the new VPS setup.

aaPanel_Kern

It is recommended install top or htop directly in the open-webui container for observation
The aapanel proxy container uses nginx to connect to the corresponding port.

PTYNETWORK

There is another image of open webui that is using GPU, could you add because just there is the default

The imagen is open-webui:cuda

PTYNETWORK

Further Testing Results: High CPU Usage with Default Docker Image of Open-WebUI on aaPanel

Hello again,

I wanted to share an update based on additional testing to help further diagnose this issue.
Additional Testing in Local Lab Environment

To isolate the problem, I set up a new environment in my local lab using aaPanel without SSL, and I deployed the default Docker image of open-webui. Here are the results:

Same High CPU Usage:
    Despite running in a local lab environment without SSL, the high CPU usage issue persists.
    As soon as I upload files (especially larger ones), the CPU usage spikes, much like in my online project. Interestingly, the process seems faster on the local setup, but it still reaches high CPU levels.

Environment Details:
    This time, I used a completely fresh aaPanel installation with the default configuration and no SSL or reverse proxy.
    The issue remains specific to aaPanel, as the same Docker image (open-webui) doesn’t have these problems in other environments (Ubuntu + Docker without aaPanel).

Summary of Findings

It appears that the high CPU usage issue with open-webui on Docker persists regardless of SSL settings or network environment, suggesting that it might be related to how aaPanel handles Docker container processes or interacts with them.
Next Steps and Request for Guidance

Given that this issue is reproducible in different aaPanel environments, are there any specific Docker or aaPanel configurations that might reduce CPU usage?

If anyone else has encountered similar issues or has insights into optimizing aaPanel’s interaction with Docker containers, I’d greatly appreciate any advice.

Thank you!

aaPanel_Kern

Hello, how can we download this container for testing?

PTYNETWORK

Hello, thank you for your continued support and willingness to investigate this further!

To facilitate testing, I can provide access to the Docker container in two possible ways. Please let me know which option is more convenient for you:
Option 1: Provide Access to a Docker Image Repository

I can upload the Docker image of open-webui to a public repository like Docker Hub or a private repository if you prefer.
Once uploaded, I can share the repository link with you for easy download and testing on your end.
This option allows you to deploy and configure the container independently.

Option 2: Provide Temporary Access to My Server

Alternatively, I can provide you with temporary access to my server where aaPanel and the open-webui container are already configured.
This option would allow you to observe the issue directly in my setup, including all configurations and resource usage.
If this approach is preferred, please advise on any specific access requirements or security protocols you’d like me to follow.

I’m happy to proceed with whichever method works best for you. Please let me know, and I’ll set it up accordingly!

Thank you!

aaPanel_Kern

Hi, can you give us your aapanel information? The server makes a snapshot backup first, if possible, please send it to kern@aapanel.com.
It is recommended to fill in the following
Post link:
SSH IP address, account password and port:
aapanel login link address and account password:
Detailed problem description:

No post link will not be able to know which user's information is, and the problem will not be processed

PTYNETWORK

Hello,

I have sent an email with the requested access details to kern@aapanel.com. This includes the aaPanel login link, SSH access, and a detailed description of the issue along with steps to reproduce it.

Thank you for your assistance. Please let me know if there’s anything else you need!

aaPanel_Kern

I don't have such a big file at the moment. Can you send one to me?
It may also be related to the problem with BT-Task. Did the same problem exist before October 29?

PTYNETWORK

you can check download any dataset here to test all of them .csv .
https://www.kaggle.com/datasets
The Issue always is with files zose more than 6 mb, 1 mb up to 5 mb take longer time uploading.

Yes this exist before October 29

aaPanel_Kern

Hello, thank you for the file you provided. I uploaded a 370KB file and it still consumes the same amount of CPU.
This should be occupied when analyzing the file, which should be normal

PTYNETWORK

This happens even in my own lab or in a dedicated server using vps.

Have you tried by yourself testing the same tool open webui?

aaPanel_Kern

I'm not testing it at the moment. It is normal for high CPU usage for analysis. Generally, GPU is used for AI.

PTYNETWORK

That doesnt matter because I tested the default image container and is the same, I think so is not normal, because the docker image container that I deploy on the server with ollama is working fine, just the 2 vps external to my own project are facing this issue with aapanel being default docker open webui contanair or cuda that is using GPU code, the same result high usage of cpu to analyze the file that shouldnt happens when is uploading so this issue just is happening with docker aapanel.

My own environment that I am using Ubuntu Server 24.04 with and docker image open webui gpu or default image without using gpu is running fine and uploading any file without facing error.

When I saw this issue well this can happens because I had just 2 core but I increased and add 6 core more (Total 8 cores )to the vps where is running aapanel and that is not possible that takes 700% usage of the proccessors.

Take note my ollama server is running on my dedicated server the open webui connected via api with ollama server, the image docker is just handling the front end that doesnt affect ollama server it all and storage any file on the website.

PTYNETWORK

sorted out.