Function Detail (Fast and Highly Efficient Communication)
Transport Selection#
Various transmission methods are available to suit different environments and purposes by combining the following connection methods and protocols.
Connecting Method
- Single Connection Mode
- Multiple Connection Mode
Available Protocols
- TCP
- HpFP (in UDP transport developed by us)
- WebSocket (SSL/TLS)
- WebSocket (no SSL/TLS)
In Single Connection Mode, the software creates a session for a single connection.
In Multiple Connection Mode, it creates multiple sessions for a single connection. This mode is useful for making file transfers on wide band networks over 10Gbps up to 100Gbps since the performance of the single connection is up to 20Gbps or less.
Selectable Connection Methods and Suitable Environments
| Connection method | Protocol | Description | Effective environment |
|---|---|---|---|
| Single-connection | TCP | General TCP communication | Environments having low latency and low packet loss |
| HpFP（UDP） | house-developed communication method | Environments having high latency and high packet loss | |
| WebSocket (SSL/TLS, cleartext) | WebSocket communication method | Environments having both of low latency and low packet loss, broadband networks or restriction that only HTTP/HTTPS communication is permitted | |
| Multiple-connection | TCP | General TCP communication | Environments having both of latency and low packet loss or broadband networks |
| HpFP（UDP） | house-developed communication method | Environments having both of high latency and high packet loss or broadband networks | |
| WebSocket(SSL/TLS, cleartext) | WebSocket communication method | Environments having both of latency and low packet loss, broadband networks or restriction that only HTTP/HTTPS communication is permitted |
* When choosing the TCP/WebSocket protocol, combined with RAPICOM seed, another house-developed product, achieves accelerated transfer.
Related Options
| Command | Short Name | Option Name | Description |
|---|---|---|---|
| hcp/hsync/hrm/hcp-ls/hmkdir/hpwd/hmv/hln/hchmod/hchown | hpfp | Use the HpFP protocol | |
| wss | Use the WebSocket protocol over SSL/TLS | ||
| udp | Use the UDP(HpFP) protocol (bi-port form, deprecated) | ||
| ws | Use the WebSocket protocol | ||
| ws-proxy | Set the proxy server in the WebSocket protocol | ||
| hcp/hsync | mcd | Set the number of maximum connections per session |
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | TCPListenAddress | TCP service listen address setting |
| HPFPListenAddress | HpFP service listen address | |
| UDPListenAddress | HpFP service listen address (bi-port type, deprecated) | |
| WSSListenAddress | WebSocket(SSL/TLS) service listen address | |
| WSSOptions | WSS options | |
| WSSCipherSuites | WSS TLS 1.3 Cipher Suites | |
| WSSCipherList | WSS TLS 1.2 or earlier Cipher Suites | |
| WSListenAddress | WebSocket(no SSL/TLS) service listen address | |
| ListenServiceBonding | Listen service bonding | |
| Client Configuration | WSSOptions | WSS options |
| WSSCipherSuites | WSS TLS 1.3 Cipher Suites | |
| WSSCipherList | WSS TLS 1.2 or earlier Cipher Suites |
Transfer Bundle of Files#
HCP tools can transfer a bundle of files by specifying file list or type to send together.
Related Options
| Command | Short Name | Option Name | Description |
|---|---|---|---|
| hcp | g | regex | File names recognized as regular expressions (but not in wildcard manner) |
| hcp | f | source-file | Source file list |
Congestion Control#
The following congestion control modes are prepared for each environment and purpose of use.
| Congestion Control Mode | Abbreviation | Description |
|---|---|---|
| Fair Mode | F | fairly shares the bandwidth with other protocols such as TCP/IP. |
| Modest Mode | M | emits packets only when the bandwidth has enough capacity to send. |
| Fair Fast-Start Mode | S | communicates at high speed initially but behaves as the fair mode after that. |
| Aggressive Mode | A | accelerates the throughput aggressively up to the maximum line bandwidth. |
Aggressive Mode is an experimental function.
When processing load increases by communications over 10 Gbps, the performance may drop by packet loss. Especially when using the multi-channel function, the CPU core is not evenly allocated by the RSS function of NIC, which may cause unstable throughput. In those cases, Fair mode is recommended.
Related Options
| Command | Short Name | Option Name | Description |
|---|---|---|---|
| hcp/hsync/hrm/hcp-ls/hmkdir/hpwd/hmv/hln/hchmod/hchown | hpfp-cong | Congestion control modes in the HpFP protocol |
Communication Data Compression Function#
Transfer data can be compressed by packet, which can shorten transmission time. This function works effectively in environments where data transmission speed is slower than the data compression speed. The compression level can be also chosen.
Related Options
| Command | Short Name | Option Name | Description |
|---|---|---|---|
| hcp | z | compress | Compress transferred data blocks |
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | HeaderCompress | Header compression (reserved) |
| ContentCompress | Content compression (reserved) | |
| Client Common Configuration | CompressLevel | Compress level |
| HeaderCompress | Header compression | |
| ContentCompress | Content compression |
Data Flow Control#
Various data flow controls allow for the best tuning to each environment.
Bandwidth control
Shaping bandwidth makes the data-transfer rate stable.
This function limits the bandwidth between the TCP/HpFP(UDP) layer and the application layer. Therefore, the application layer gets limited. However, the Communication buffer and the transmission control on the transport layer are not affected.
Servers can set the configuration both on the system and on each connection.
With over 5Gbps in each setting, the bandwidth gets unlimited, which results in no shaping.
Related Options
| Command | Short Name | Option Name | Description |
|---|---|---|---|
| hsync | bwlimit | Limit socket I/O bandwidth |
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | MaxTotalReceiveRate | Maximum total receiving rate (entire system) |
| MaxTotalSendRate | Maximum total sending rate (entire system) | |
| MaxReceiveRatePerConnection | Maximum receiving rate (per connection) | |
| MaxSendRatePerConnection | Maximum sending rate (per connection) | |
| Client Common Configuration | MaxReceiveRate | Max receiving rate (by session) |
| MaxSendRate | Max sending rate (by session) |
File lock
This is a file lock function that the file gets locked and inaccessible while the file data is being read or written.
File lock is a system that temporarily confines the users and processes that can access to files.
In the process, after the file gets locked, the file is read or written. Advisory lock is supported on the Linux platform. This function is available between this software and also other software that uses the same file lock system.
When a hung-up happens in the network file system, such as NFS(Network File System), or an error occurs when getting a file lock, this function has to be disabled. It is also requested to make sure the destination in the hcp commands is not input double.
And configurations of the number of maximum trials and the interval time on the file lock are also provided.
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | FileLock | Use file lock |
| FileLockTrials | Number of trials to lock files | |
| FileLockTrialInterval | The trial interval (in seconds) | |
| Client Common Configuration | FileLock | Use file lock |
| FileLockTrials | Number of trials to lock files | |
| FileLockTrialInterval | The trial interval (in seconds) |
Creating Temporary File to Save (atomically file saving)
This function performs a two-step file saving using a temporary file on writing transferred data to a file.
It's for writing dat to files in safety.
On the Linux versions, the write access right in the final destination is checked before creating temporary files. With the resume function (-r option), files transferred part of the way are transferred from the beginning.
The overwrite-als-temp-file option of the hcp command requests to overwrite a temporary file which already exists. Without this option, the transferring file goes to the transfer error when a temporary file already exists. As long as AtomicLikeSavingRejectOverwriteRequest is set to yes in the server setting, even though this option is on, overwriting a temporary file which already exists is rejected.
The configuration of a threshold in file size to enable saving files atomically is provided. The saving atomically does not cover files under the threshold. "0 bytes" means all files are covered.
Related Options
| Command | Short Name | Option Name | Description |
|---|---|---|---|
| hcp | overwrite-als-temp-file | Atomically overwrite a temporary file |
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | AtomicLikeSaving | Atomically file saving |
| AtomicLikeSavingThreshold | Threshold for atomically file saving | |
| AtomicLikeSavingRejectOverwriteRequest | Reject requests to overwriting temporary files which already exist when atomically file saving | |
| Client Configuration (hcp) | AtomicLikeSaving | Atomically file saving |
| AtomicLikeSavingThreshold | Threshold for atomically file saving |
Data Buffer Configuration
HCP tools setups a data buffer in a part of the general main memory for buffering transferred data. This is a function to configure size of the data buffer.
Its configuration on the server side includes a total size on the system and a size per session.
And configurations of a sending buffer size on TCP (TCPServiceSocketSendBuffer) and an extension buffer size of HpFP (UDPServiceExtensionBufferSize) are also provided.
You need the option of TCPServiceSocketSendBuffer to make a performance tuning of TCP on 100G environment. No need to use in ordinary cases.
Each HpFP transport session extends a buffer holding transferred data up to a size specified by hpfp_sndbuf or hpfp_rcvbuf on HPFPListenAddress or UDPListenAddress. '0' indicates disabling this buffer extension and the initial size before extending is 1MB.
Workarounds when a processe is killed by Linux OOM (Out Of Memory) Killer
As an OS mechanism, Linux monitors memory consumption trends of processes and terminates (KILL signal) processes which run out of memory. When you set the following buffer size options, e.g, MaxTotalBufferSize or MaxBufferSize, in this software to values large than system memory, Linux OOM Killer might kill the process. For fixing it you have to specify smaller buffer size values for the options or extending the system memory.
Related Options
| Command | Short Name | Option Name | Description |
|---|---|---|---|
| hcp/hsync/hrm/hcp-ls/hmkdir/hpwd/hmv/hln/hchmod/hchown | hpfp-sndbuf | Set the sending buffer size for the HpFP protocol | |
| hpfp-rcvbuf | Set the receiving buffer size for the HpFP protocol |
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | HPFPListenAddress | HpFP service listen address |
| UDPListenAddress | HpFP service listen address (bi-port type, deprecated) | |
| MaxTotalBufferSize | Maximum buffer allocation size (entire system) | |
| MaxBufferSizePerConnection | Maximum buffer allocation size (per connection) | |
| TCPServiceSocketSendBuffer | TCP sending buffer | |
| UDPServiceExtensionBufferSize | HpFP service extension buffer size | |
| Client Common Configuration | MaxBufferSize | Maximum buffer allocation size |
| UDPTransportExtensionBufferSize | HpFP transport buffer size extension | |
| TCPTransportSocketSendBuffer | TCP sending buffer |
Limitation File Size to Transfer
This is a function to set maximum sizes of files to transfer.
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | MaxReceiveFileSize | Maximum receiving file size |
| MaxSendFileSize | Maximum sending file size | |
| Client Common Configuration | MaxReceiveFileSize | Maximum receiving file size |
| MaxSendFileSize | Maximum sending file size |
Communication Message Size Control
This is a function to configure sizes of transfer data blocks made from payloads of files.
HCP tools evaluates consumption of bandwidth after starting transfers. And it increases and decreases sizes of header blocks including messages, e.g. file requests, and content blocks including a payload data of a file subject to the evaluation.
You can set options for that mechanisms, InitHeaderBlockSize for a initial size of the header blocks, InitContentBlockSize for a initial size of the content blocks, MaxHeaderBlock for a maximum size of the header blocks and MaxContentBlockSize for a maximum size of the content blocks.
And a configuration of MSS (Maximum Segment Size) for HpFP protocol is also provided for the frame size control on transport.
Related Options
| Command | Short Name | Option Name | Description |
|---|---|---|---|
| hcp/hsync/hrm/hcp-ls/hmkdir/hpwd/hmv/hln/hchmod/hchown | hpfp-mss | Set MSS for the HpFP protocol |
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | InitHeaderBlockSize | Initial header block size |
| InitContentBlockSize | Initial content block size | |
| MaxHeaderBlockSize | Maximum header block size | |
| MaxContentBlockSize | Maximum content block size | |
| MaxRequestFileEntryAtOnce | Max file entry request | |
| Client Common Configuration | InitHeaderBlockSize | Initial header block size |
| InitContentBlockSize | Initial content block size | |
| MaxHeaderBlockSize | Maximum header block size | |
| MaxContentBlockSize | Maximum content block size | |
| MaxRequestFileEntryAtOnce | Max file entry request |
Disk I/O Speed Control
This is a function to set maximum speeds of writing data to a disk and reading from it on file transfers.
In some environment, long time running of reading and writing might make overheat on SSDs (especially the writing side). So this option is useful for avoiding it.
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | MaxReadRatePerConnection | Disk I/O reading rate per session |
| MaxWriteRatePerConnection | Disk I/O writing rate per session | |
| Client Configuration (hcp) | MaxReadRate | Disk I/O reading rate per session |
| MaxWriteRate | Disk I/O writing rate per session |
String Code Conversion#
This is a function to avoid character corruption by converting strings among string codes.
Transport Character Encoding Negotiation
This is a function to determine a character encoding for strings to be transferred over networks.
Encodings to use are supposed to be specified in configuration files of the client and server. The encodings will be applied to strings, e.g. file paths exchanged between the server and client. HCP tools determines an actual encoding to use by a way that it compares configurations of the server and client and determines an encoding defined in common.
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | TransportCharEncoding | Transport character encoding |
| Client Common Configuration | TransportCharEncoding | Transport character encoding |
Host Character Encoding Support
This is a function to convert strings in a string code on the operating system (host) to strings in another string code on the software processing of HCP tools.
This function avoids character corruption by setting a string code used on the host operating system. For example, that string conversion will be applied to file paths to convert them to the software internal representation.
Related Configurations
| Category | Configuration Name | Description |
|---|---|---|
| Server Configuration | HostEncoding | Host character encoding |
| Client Common Configuration | HostEncoding | Host character encoding |