yumoqing/pipeline_core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
pipeline_core/skills_library/all/apppublic-python-module/SKILL.md

7.2 KiB
Raw Blame History

name description author tags
apppublic-python-module Comprehensive guide to using the appPublic Python utility module for configuration, logging, networking, cryptography, and data processing Hermes Agent
python
utilities
configuration
logging
networking
cryptography
data-processing

appPublic Python Module Guide

Overview

appPublic is a comprehensive utility library (version 5.5.0) designed to provide a wide range of common functionality for Python applications. It requires Python >=3.8 and includes utilities for configuration management, logging, HTTP clients, cryptography, networking, data processing, and more. The module serves as a foundational toolkit that eliminates the need to repeatedly implement common patterns and utilities across different projects.

Key Modules and Functionality

Configuration Management

  • Config.py: Provides singleton-based configuration management using INI files with support for custom object types (Node, DictObject)
  • jsonConfig.py: JSON-based configuration loader with template variable substitution support via ArgsConvert
  • JsonObject: Extends DictObject to load and manipulate JSON configuration files with namespace support

Logging and Monitoring

  • mylog.py: Custom logging system with categorized logging levels (SYSError, SYSWarn, APPError, APPWarn, APPInfo, DEBUG1-5)
  • timecost.py: Performance monitoring and timing utilities
  • LogMan: Centralized log manager supporting multiple loggers and categories

HTTP and Networking

  • http_client.py: Enhanced HTTP client wrapper around requests.Session with automatic session management, custom response handlers, and error handling (NeedLogin, InsufficientPrivilege, HTTPError exceptions)
  • proxy.py: SOCKS5 proxy configuration utilities for socket-level proxying
  • sshx.py: Advanced SSH client with jump server support, async context managers, and connection pooling using asyncssh
  • zmqapi.py: ZeroMQ integration with async/await support for PUB/SUB messaging patterns, proxy services, and request-response patterns
  • udp_comm.py: UDP communication utilities
  • port_forward.py: TCP/UDP port forwarding capabilities

Cryptography and Security

  • RSAutils.py: RSA encryption/decryption utilities using PyCryptodome with PKCS1_OAEP and PKCS1_V1_5 support
  • rsawrap.py: Additional RSA wrapper functionality
  • rc4.py: RC4 stream cipher implementation

Data Processing and Utilities

  • dictObject.py: Dictionary-to-object conversion class that allows attribute-style access to dictionary keys with JSON serialization support
  • unicoding.py: Unicode string handling and encoding conversion utilities with fallback mechanisms
  • exceldata.py: Excel file processing capabilities (requires xlrd, xlwt)
  • CSVData.py: CSV data handling utilities
  • dataencoder.py: Data encoding/decoding utilities
  • datamapping.py: Data transformation and mapping utilities

Time and Date Utilities

  • timeUtils.py: Comprehensive date/time utilities including:
    • Date difference calculations
    • Current timestamp generation
    • Month/day boundary detection
    • Various date/time formatting functions
    • Leap year handling

System and Process Management

  • tworkers.py: Thread and worker management utilities
  • asynciorun.py: Asyncio execution helpers
  • objectAction.py: Object lifecycle and action management
  • FiniteStateMachine.py: Finite State Machine implementation with state transition management

Internationalization and Localization

  • localefunc.py: Locale-specific functionality
  • MiniI18N.py: Minimal internationalization support
  • country_cn_en.py: Country name mappings between Chinese and English

Development and Debugging Tools

  • myImport.py: Dynamic module import utility for nested module paths
  • myTE.py: Template engine utilities
  • argsConvert.py: Template variable substitution (e.g., [variable] syntax)
  • testdict.py: Testing utilities for dictionary operations

Specialized Utilities

  • uniqueID.py: Unique identifier generation
  • ObjectCache.py: Object caching mechanisms
  • outip.py / uni_outip.py: External IP address detection
  • ipgetter.py: IP address retrieval utilities
  • wav.py: Audio/WAV file utilities
  • genetic.py: Genetic algorithm utilities
  • myjson.py / jsonIO.py: Enhanced JSON handling

Usage Patterns

Configuration Loading

from appPublic.jsonConfig import JsonConfig
config = JsonConfig('config.json', NS={'env': 'production'})
value = config.some_setting

HTTP Client Usage

from appPublic.http_client import Http_Client
client = Http_Client()
response = client._webcall('https://api.example.com', method='GET', params={'key': 'value'})

Logging

from appPublic.mylog import MyLog
logger = MyLog('/path/to/log')
logger('Application started')

RSA Cryptography

from appPublic.RSAutils import newkeys, encrypt, decrypt
public_key, private_key = newkeys(2048)
encrypted = encrypt(b'message', public_key)
decrypted = decrypt(encrypted, private_key)

ZeroMQ Messaging

from appPublic.zmqapi import zmq_subscribe
# Async subscription pattern
await zmq_subscribe('topic_key', callback_function)

SSH Operations

from appPublic.sshx import SSHServer
server_config = {'host': 'example.com', 'username': 'user', 'password': 'pass'}
ssh_server = SSHServer(server_config)
async with ssh_server.get_connector() as conn:
    result = await conn.run('ls -la')

Dependencies

The module requires the following dependencies:

  • pillow, qrcode, xlrd, xlwt (data processing)
  • bs4 (BeautifulSoup for HTML parsing)
  • numpy (numerical operations)
  • rsa, cryptography, bcrypt (cryptography)
  • aiohttp, asyncio, aiohttp_socks (async HTTP and networking)
  • requests (synchronous HTTP)
  • jinja2 (templating)
  • pyzmq (ZeroMQ messaging)
  • asyncssh (SSH client)
  • psutil (system monitoring)
  • ujson, brotli (performance utilities)
  • nanoid (unique ID generation)
  • eventpy (event handling)

Design Philosophy

appPublic follows several key design principles:

  1. Singleton Pattern: Many core utilities (Config, JsonConfig) use singleton decorators to ensure single instances
  2. Dictionary-Object Hybrid: DictObject provides seamless transition between dictionary and object paradigms
  3. Async-First: Modern modules support async/await patterns alongside traditional synchronous code
  4. Error Handling: Comprehensive exception hierarchy for different error scenarios
  5. Extensibility: Modular design allows easy extension and customization

Target Use Cases

  • Web application backends requiring robust configuration and logging
  • Microservices with messaging requirements (ZeroMQ, HTTP APIs)
  • System administration tools needing SSH and network utilities
  • Data processing pipelines requiring Excel/CSV handling
  • Security-sensitive applications needing cryptography utilities
  • IoT and embedded systems requiring lightweight, efficient utilities

This comprehensive utility library eliminates boilerplate code and provides production-ready implementations of common patterns, making it ideal for rapid application development while maintaining enterprise-grade reliability.