Commit Graph
100 Commits
Author SHA1 Message Date
tomasz.sowa c7adffd9fd sort a makedepend output
Makefile.dep is sorted now - it's better when making a diff to the previous version.
2024-09-22 15:24:05 +02:00
tomasz.sowa 98e2d5d0ac add a clangd and qtcreator make targets
Add a clangd make target:
$ make clangd
This installs a .clangd file with configuration (includes/macros/compiler flags)
for the clandg language server.

Add a qtcreator make target:
$ make qtcreator
This installs a .qtcreator directory with QTCreator IDE project files.
In the QTCreator you can open a project by selecting the .qtcreator/pikotools.creator
file.

You can use 'make qtcreator' again to refresh .qtcreator/pikotools.files,
the rest of the project files will not be modified.
2024-09-22 11:30:04 +02:00
tomasz.sowa 54b8ebfd8a add a remove_white(...) method 2024-08-06 20:13:31 +02:00
tomasz.sowa 6165a2ece3 add a new utf8_to_output_function() method
add such a method:
template<typename OutputFunction>
bool utf8_to_output_function(const char * utf8, OutputFunction output_function, int mode = 1);
2024-06-19 22:54:16 +02:00
tomasz.sowa f85f1dade5 improve the Space text convertion methods
Read the whole character from a multibyte string (as int/char32_t) and
then check if it needs to be escaped. Also don't use a tmp stream object
when serializing between wide/char strings.

while here:
- add try_esc_to_space(...) global function
- add wide_to_output_function(const wchar_t * str, size_t len, OutputFunction output_function, int mode)
- add wide_to_output_function(const wchar_t * str, OutputFunction output_function, int mode)
2024-06-19 04:46:00 +02:00
tomasz.sowa c0838de3a4 use a char32_t character in the base Stream class
Add an operator<<(char32_t) to the Stream class, char32_t will be used
as a main character instead of a wchar_t (this is needed on systems
where sizeof(wchar_t) is equal to 2).

while here:
- add to utf8:
  size_t wide_to_int(const Stream & stream, size_t stream_index, int & res, bool & correct)
  template<typename StreamType, typename OutputFunction> bool wide_to_output_function(StreamType & buffer, OutputFunction output_function, int mode = 1)
  template<typename OutputFunction> bool wide_to_output_function_by_index(const Stream & stream, OutputFunction output_function, int mode)
- add to convert/misc:
  bool try_esc_to_tex(char32_t c, pt::Stream & out)
  bool try_esc_to_html(char32_t c, pt::Stream & out)
2024-05-31 23:11:11 +02:00
tomasz.sowa 2689c9fece update utf8 functions comments
while here:
- rename pt::utf8_check_range(...) -> pt::is_correct_unicode_char(...)
2024-05-31 00:23:43 +02:00
tomasz.sowa 450c5d55e9 leavy only utf8.h and utf8.cpp
Remove utf8_private.h, utf8_private.cpp and utf8_templates.h
and move their methods to utf8.h/utf8.cpp.
2024-05-30 21:20:25 +02:00
tomasz.sowa aacb1f43ae add some utf8 converting methods
add new methods:
- bool int_to_stream(int c, pt::Stream & stream);
- template<typename OutputFunction>
  bool utf8_to_output_function(const Stream & stream, OutputFunction output_function, int mode = 1);
- template<typename StreamIteratorType, typename OutputFunction>
  bool utf8_to_output_function(StreamIteratorType & iterator_in, const StreamIteratorType & iterator_end, OutputFunction output_function, int mode = 1);
- template<typename StreamType, typename OutputFunction>
  bool wide_to_output_function(StreamType & buffer, OutputFunction output_function, int mode = 1);

make some methods public:
- size_t wide_to_int(const wchar_t * wide_string, size_t string_len, int & z, bool & correct)
- size_t wide_to_int(const wchar_t * wide_string, int & z, bool & correct)

rename and make some methods public:
- template<typename OutputFunction>
  utf8_to_wide_generic(const char * utf8, size_t utf8_len, OutputFunction convert_function, int mode) -> utf8_to_output_function(...)

while here:
- fix: correctly convert characters in Log::put_multiline_generic()
2024-05-30 20:19:04 +02:00
tomasz.sowa 5fd17175c1 add a TextStreamBase<>::operator<<(morm::Model & model) 2024-05-29 16:37:53 +02:00
tomasz.sowa f35e2122ed (HtmlParser): rename ItemParsedListener to Listener
while here:
- add a new callback method: bool should_remove(Item &)
2024-04-17 10:39:06 +02:00
tomasz.sowa f02dd1093a fix(HtmlParser): correctly remove an item from the space struct when requested from a callback
while here:
- implement the removing algorithm for the compact_mode
2024-04-16 09:35:47 +02:00
tomasz.sowa 90915a7209 add a KeyValueParser for parsing simple key/value strings 2023-11-07 03:54:34 +01:00
tomasz.sowa df24be199d add QT Creator files to .gitignore 2023-10-22 23:36:47 +02:00
tomasz.sowa 21614a5309 fix: memory leak in the HTMLParser when a compact mode was used 2023-10-18 18:33:52 +02:00
tomasz.sowa 57f49cdcb6 fix: memory leak in the SpaceParser 2023-10-18 16:51:52 +02:00
tomasz.sowa e94589d6b5 add a Date::CompareDate(...) and Date::CompareTime(...) methods 2023-09-07 04:41:41 +02:00
tomasz.sowa 09215ef5f2 add Space::to_date(...) methods 2023-07-18 18:22:13 +02:00
tomasz.sowa 2c4bfe085b add == and != operators to the TextStreamBase<> class 2023-07-14 09:07:57 +02:00
tomasz.sowa 172c2fcee7 add to_str(...) methods to the TextStreamBase<> class
add such methods:
bool to_str(char * str, size_t max_buf_len) const;
bool to_str(wchar_t * str, size_t max_buf_len) const;
2023-07-14 07:42:09 +02:00
tomasz.sowa 78d31861de add some wide/utf8 convertion methods
add following methods:
size_t int_to_wide(int c, wchar_t * res, size_t max_buf_len);

template<typename StreamIteratorType>
bool utf8_to_wide(StreamIteratorType & iterator_in, const StreamIteratorType & iterator_end, wchar_t * out_buffer, size_t max_buffer_len, int mode = 1, bool * was_buffer_sufficient_large = nullptr);

template<typename StreamType>
bool utf8_to_wide(const StreamType & stream, wchar_t * out_buffer, size_t max_buffer_len, bool * was_buffer_sufficient_large = nullptr, int mode = 1);

template<typename StreamType>
bool wide_stream_to_utf8(StreamType & buffer, char * utf8, std::size_t max_buffer_size, bool * was_buffer_sufficient_large = nullptr, int mode = 1);
2023-07-14 07:41:14 +02:00
tomasz.sowa 7e92b5d9d7 add HTMLParser::parse_xml(...) methods 2023-07-04 22:58:43 +02:00
tomasz.sowa cbaf57bec3 add some constants to the Date class 2023-07-04 22:57:50 +02:00
tomasz.sowa 987d9c845c declare esc_to_csv() method with a wstring 2023-05-27 18:18:00 +02:00
tomasz.sowa 96a3a564cf add a virtual dctor to BaseParser() 2022-12-23 04:38:03 +01:00
tomasz.sowa 379adf6a69 allow to parse a time decimal fraction in ParseTime() method
while here:
- let ParseDate() is able to parse such formats: "20081012" (without a separator)
  and without the month or day e.g: "2008" or "200810"
- let ParseTime() is able to parse a time without separators, e.g.:
  "141030", or "1410" or just "14"
- let Parse(...) method use ParseDate() and ParseTime()
  this will parse a format similar to ISO 8601
2022-12-23 02:15:11 +01:00
tomasz.sowa 3b3c04b85d fix: rename Toul -> to_ul in PatternReplacer 2022-11-16 16:14:16 +01:00
tomasz.sowa b3137a7607 rename functions for converting strings to integers to snake case
while here:
- add some functions taking std::string/std::wstring
2022-11-14 03:20:17 +01:00
tomasz.sowa f97c06d441 add a check_time_zone parameter when parsing a date 2022-10-22 16:26:14 +02:00
tomasz.sowa e501a3f4a3 remove FileLog::synchro_lock() and FileLog::synchro_unlock() 2022-09-01 07:32:48 +02:00
tomasz.sowa ce0348b2d7 add to Space methods which takes a Stream as an argument
- Space::set(const Stream & stream)
- Space::add(const Stream & stream)
- Space::add(const wchar_t * field, const Stream & stream)
- Space::add(const std::wstring & field, const Stream & stream)
2022-08-20 00:26:12 +02:00
tomasz.sowa 7eba07a439 fix(Space): increment value object iterator in get_space_nc 2022-08-10 12:40:46 +02:00
tomasz.sowa 663233fe2a let all utf8/wide functions can be available just by including utf8/utf8.h
while here:
- remove utf8/utf8_stream.h, now we only need utf8/utf8.h to include
- add some new methods for converting from a utf8 stream to wide stream/string
- do some improvements in TextStream:
  - don't use temporary objects to convert utf8/wide
  - add put_stream() which takes TextStreamBase<> as its argument
    (uses an iterator instead of get_char() for reading)
  - let operator<<(const Space & space) serialize to json and not to Space
2022-07-30 03:31:18 +02:00
tomasz.sowa 84e9e6f98f add methods to Space that take a pointer to a string along with the length
Space::Space(const char * str, size_t len)
Space::Space(const wchar_t * str, size_t len)
Space::set(const char * str, size_t len)
Space::set(const wchar_t * str, size_t len)
Space::add_to_table(const char * val, size_t len)
Space::add_to_table(const wchar_t * val, size_t len)
Space::add(const wchar_t * field, const char * val, size_t len)
Space::add(const wchar_t * field, const wchar_t * val, size_t len)
Space::add(const std::wstring & field, const char * val, size_t len)
Space::add(const std::wstring & field, const wchar_t * val, size_t len)
2022-07-30 03:12:38 +02:00
tomasz.sowa 9a596dd097 fix: return a correct value from Log::size and Log::capacity 2022-07-30 02:45:19 +02:00
tomasz.sowa aa97fe2811 add methods for trimming \r\n from the end of a string
add:
void trim_last_new_lines(std::string & str, bool check_carriage_return_too = true);
void trim_last_new_lines(std::wstring & str, bool check_carriage_return_too = true);
2022-07-30 02:43:29 +02:00
tomasz.sowa d13c10c604 add methods for converting from hex string to bytes
add to convert/text.h:
template<typename HexStringPointerType, typename BytesStringType>
bool hex_string_pointer_to_bytes(const HexStringPointerType * hex_string, BytesStringType & bytes, bool clear_bytes = true);

template<typename HexStringType, typename BytesStringType>
bool hex_string_to_bytes(const HexStringType & hex_string, BytesStringType & bytes, bool clear_bytes = true);
2022-07-26 05:14:35 +02:00
tomasz.sowa a524dfa2a7 add Space::to_float(...), to_double(...) and to_long_double(...) methods 2022-07-08 21:59:39 +02:00
tomasz.sowa 4b68eba6d2 add LICENSE file 2022-07-03 11:02:16 +02:00
tomasz.sowa b81daf9fb6 set 2-Clause BSD licence in *.cpp files 2022-06-30 13:44:21 +02:00
tomasz.sowa 74230d667b change headerfile_picotools_* macros to headerfile_pikotools_* 2022-06-30 12:45:08 +02:00
tomasz.sowa dad8042c41 add pikotools/version.h file 2022-06-30 12:44:06 +02:00
tomasz.sowa cadba907b2 change licence from 3-Clause BSD to 2-Clause BSD 2022-06-30 12:09:22 +02:00
tomasz.sowa 4933378ed6 make depend 2022-06-26 05:40:44 +02:00
tomasz.sowa 4f07c00217 Merge branch 'api2021' 2022-06-25 17:52:42 +02:00
tomasz.sowa 44bda888b5 fix: do not unescape xml sequences in filter mode 2022-06-01 05:17:30 +02:00
tomasz.sowa 68fe25c8bf add limits when parsing a json/space format
while here:
- add column index error
- add parsing methods with pt::TextStream and pt::WTextStream arguments
2022-05-30 01:01:14 +02:00
tomasz.sowa a40bab0445 add Space::get_table_item() method 2022-05-30 00:55:38 +02:00
tomasz.sowa c3b7ab5793 add min_width parameter to methods converting int to string 2022-05-28 06:06:32 +02:00
tomasz.sowa 5d2788d0d8 add Log::put_multiline() methods 2022-05-25 19:57:35 +02:00
tomasz.sowa 72c10b20fb flush logs when printing to stdout 2022-04-27 22:07:58 +02:00
tomasz.sowa 3173042229 make depend 2022-04-26 23:47:27 +02:00
tomasz.sowa 5253963c84 fix: put a white char before an opening tag in tree mode if it was in the source html 2022-02-08 16:34:54 +01:00
tomasz.sowa 0100c7e453 fix: check correctly for new lines when filtering html 2022-02-08 14:52:50 +01:00
tomasz.sowa ac3c59323b add methods: try_esc_to_json(wchar_t val, stream) try_esc_to_xml(...) try_esc_to_csv(...)
Those methods return true if the val character was escaped and put
to the out stream. If the character is invalid for such a stream
they only return true without putting it to the stream.
2022-02-04 14:19:54 +01:00
tomasz.sowa 3b9b464bb7 fix: add typename keyword in TextStreamBase<> in some places 2022-02-03 19:21:22 +01:00
tomasz.sowa 6b97b1b74a fix: correctly escape json/xml/csv wide strings
A wide string was first changed to utf-8 and then escaped to json/xml/csv
which is incorrect. First should be escaped and then changed to utf-8.

Add TextStreamBase<>::iterator and TextStreamBase<>::const_interator as classes
with a method wchar_t get_unicode_and_advance(const iterator & end)
to return one character either from utf-8 stream or from wide stream.

Let TextStreamBase<>::operator<<(wchar_t v) correctly use utf-8.
2022-02-03 19:08:21 +01:00
tomasz.sowa fd1a8270cd read CDATA as an ordinary text 2022-01-18 19:36:40 +01:00
tomasz.sowa b781948f21 HTMLParser now parses correctly such entities: &amp; &lt; &gt; &quot; &apos; 2021-12-02 17:44:41 +01:00
tomasz.sowa 2dadfc0809 added: HTMLParser::ItemParsedListener listener with an item_parsed(...) method which is called when a tag is parsed by the parser 2021-11-30 16:27:27 +01:00
tomasz.sowa bb9205a55e added: Space::Space(const Date & date), Space::set(const Date & date), Space::add(const Date & date), Space::add(const wchar_t * field, const Date & date) 2021-11-05 09:27:32 +01:00
tomasz.sowa 5eff9a5f4f Space::to_bool() return true now when a string/object or table is non empty 2021-10-20 08:30:57 +02:00
tomasz.sowa c54c398828 fixed in HTMLParser: </nofilter> tag was printed 2021-10-13 00:40:55 +02:00
tomasz.sowa 17d2c0fb25 - added some converting methods: esc_to_json(...), esc_to_xml(...), esc_to_csv() (convert/misc.h)
- BaseParser: added possibility to read from TextStream and WTextStream
- HTMLParser: added filter(const WTextStream & in, Stream & out, ...) method
- added utf8_stream.h with one method:
  template<typename StreamIteratorType>
  size_t utf8_to_int(
    StreamIteratorType & iterator_in,
    StreamIteratorType & iterator_end,
    int & res,
    bool & correct)
2021-10-12 19:53:11 +02:00
tomasz.sowa 4902eb6037 fixed: in HTMLParser::CheckClosingTags() don't return immediately if stack_len is equal to 2 2021-10-03 13:22:49 +02:00
tomasz.sowa 5e4c7e9929 make depend 2021-10-02 21:01:19 +02:00
tomasz.sowa abe349be34 small refactoring in HTMLParser 2021-10-02 21:01:09 +02:00
tomasz.sowa f23cabfb2f added to HTMLParser: filter_file(...) methods for filtering from a file 2021-10-02 20:34:19 +02:00
tomasz.sowa 5b2583b566 fixed in HTMLParser: sometimes a closing item left on the stack, for stack_len < 3 there was not PopStack() called 2021-10-02 18:45:02 +02:00
tomasz.sowa 2cc9dd69a3 make depend 2021-08-12 21:53:52 +02:00
tomasz.sowa 2576eb12d1 HTMLParser: start working on xml mode
added methods:
Status parse_xml_file(const char * file_name,         Space & out_space, bool compact_mode = false, bool clear_space = true);
Status parse_xml_file(const std::string & file_name,  Space & out_space, bool compact_mode = false, bool clear_space = true);
Status parse_xml_file(const wchar_t * file_name,      Space & out_space, bool compact_mode = false, bool clear_space = true);
Status parse_xml_file(const std::wstring & file_name, Space & out_space, bool compact_mode = false, bool clear_space = true);
2021-08-10 21:56:04 +02:00
tomasz.sowa b1cc64a29b added a compact_mode option when creating a space output 2021-08-10 01:45:10 +02:00
tomasz.sowa b8a03bf852 HTMLParser: added possibility to parse html to Space class
added method: HTMLParser::parse_html(const wchar_t * in, Space & space)
2021-08-07 21:21:16 +02:00
tomasz.sowa 7fcfdac52f Space: added pretty_print parameter to some json serializing methods 2021-08-07 21:19:38 +02:00
tomasz.sowa 8c5ede5cf3 HTMLParser: for <script> and <!- (comments) we copy the content without parsing 2021-08-07 02:13:13 +02:00
tomasz.sowa fdfd0b1385 renamed: HTMLFilter -> HTMLParser 2021-08-06 17:10:19 +02:00
tomasz.sowa f6df8bc1bc HTMLFilter: added a std::vector<int> stack for a current white mode - white chars mode can be changed by such tags: <textarea>, <pre>, <script>, <nofilter> 2021-07-21 15:57:46 +02:00
tomasz.sowa c0e940c500 fixed improper new line character after <single/> items, added Item::new_line_before flag 2021-07-21 11:30:49 +02:00
tomasz.sowa 4f8ae6ce29 some work in HTMLFilter
- instead of directly using pchar pointer now we use pointers/streams from BaseParser
- removed support for putting a white char in long words: removed BreakWord(size_t break_after_) method
- changed the way how white characters are treated: added white_chars_mode(int mode) method
  mode 0: WHITE_MODE_ORIGIN
  mode 1: WHITE_MODE_SINGLE_LINE
  mode 2: WHITE_MODE_TREE
2021-07-20 20:48:01 +02:00
tomasz.sowa 7ce07c57f5 added a base class for parsers: BaseParser (convert/baseparser.h|cpp)
there are methods for reading from string/files there
  those methods were moved from SpaceParser and CSVParser
fixed: CSVParser didn't set input_as_utf8 flag
2021-07-17 14:38:22 +02:00
tomasz.sowa 2a3f43c5c3 added BBCODEParser (html/bbcodeparser.h|cpp) - copied from winix project 2021-07-17 13:54:03 +02:00
tomasz.sowa bdb2616f32 added: HTMLFilter (html/htmlfilter.h|cpp) - copied from winix project 2021-07-17 13:35:10 +02:00
tomasz.sowa 6c41e0a803 Merge branch 'api2021' 2021-07-06 22:45:54 +02:00
tomasz.sowa 1e5598cde1 added to Date: SerializeMonthAsRoman(Stream & out, int month) - serialize month in Roman numerals
added a param: 'bool roman_month' to some serialize methods
2021-07-06 21:44:04 +02:00
tomasz.sowa 198945c97b PatternReplacerBase: to_string() changed to to_str() 2021-07-06 21:42:42 +02:00
tomasz.sowa 34f1fc04cf added Space::remove(size_t table_index) for removing a table item
fixed: pretty printing for Space format
2021-06-29 23:25:31 +02:00
tomasz.sowa 8997284b16 added trim(...) functions to convert/text.h
void trim_first_white(std::string & str, bool check_additional_chars = true, bool treat_new_line_as_white = true);
void trim_first_white(std::wstring & str, bool check_additional_chars = true, bool treat_new_line_as_white = true);

void trim_last_white(std::string & str, bool check_additional_chars = true, bool treat_new_line_as_white = true);
void trim_last_white(std::wstring & str, bool check_additional_chars = true, bool treat_new_line_as_white = true);

void trim_white(std::string & str, bool check_additional_chars = true, bool treat_new_line_as_white = true);
void trim_white(std::wstring & str, bool check_additional_chars = true, bool treat_new_line_as_white = true);

void trim_first(std::string & str, wchar_t c);
void trim_first(std::wstring & str, wchar_t c);

void trim_last(std::string & str, wchar_t c);
void trim_last(std::wstring & str, wchar_t c);

void trim(std::string & str, wchar_t c);
void trim(std::wstring & str, wchar_t c);
2021-06-29 23:23:35 +02:00
tomasz.sowa e31ef3c6c4 make depend 2021-06-27 22:34:05 +02:00
tomasz.sowa e0d6e7fcb1 added to Space:
Space & get_add_space(const wchar_t * field);
Space & get_add_space(const std::wstring & field);
2021-06-27 15:58:53 +02:00
tomasz.sowa 009e240a8d fixed some memory leaks in Space, pointers in tables and objects were not correctly 'deleted', affected methods:
set_empty_table()
set_empty_object()
add(const wchar_t * field, Space && space)
copy_value_object(const Value & value_from)
copy_value_table(const Value & value_from)
initialize_value_object_if_needed(ObjectType && obj)
initialize_value_table_if_needed(TableType && tab)
add_generic(const wchar_t * field, const ArgType & val)
2021-06-27 15:41:38 +02:00
tomasz.sowa 4a1630b1ea removed support for so called child objects from Space (this was an old feature of Space struct, now not needed)
Space::get_object_field(...) renamed to Space::get_space(...)
2021-06-26 22:56:12 +02:00
tomasz.sowa 8ec9350d52 added two functions to utf8:
template<typename StreamType> bool utf8_to_wide(const Stream & stream, StreamType & res, bool clear = true, int mode = 1);
template<typename StreamType> bool wide_stream_to_utf8(const Stream & stream, StreamType & utf8, bool clear = true, int mode = 1);

these functions are moved from TextStreamBase
2021-06-25 19:10:01 +02:00
tomasz.sowa 792057a869 make depend 2021-06-24 21:18:48 +02:00
tomasz.sowa 4d9f5f6c55 Log class has the Stream class as a base class now
- implemented some missing operators<<(...)
- removed Manipulators: l1, l2, l3, l4, lend, lsave
- PascalCase to snake_case in Log

added to Stream:
  virtual bool is_char_stream() const = 0;
  virtual bool is_wchar_stream() const = 0;
  virtual char get_char(size_t index) const = 0;
  virtual wchar_t get_wchar(size_t index) const = 0;
  virtual Stream & operator<<(const Stream & stream) = 0;
2021-06-24 20:52:48 +02:00
tomasz.sowa 2b6789754f implemented pretty printing in Space::serialize_to_json_stream(StreamType & str, bool pretty_print, int level) 2021-06-23 21:54:34 +02:00
tomasz.sowa 3c0b59e115 added to Space: long double to Space::Value and methods for converting from/to long double
added global methods for converting float/string double/string and long double/string (convert/double.h|cpp):
      float to_float(const char * str, const char ** after = nullptr);
      float to_float(const wchar_t * str, const wchar_t ** after = nullptr);
      double to_double(const char * str, const char ** after = nullptr);
      double to_double(const wchar_t * str, const wchar_t ** after = nullptr);
      long double to_long_double(const char * str, const char ** after = nullptr);
      long double to_long_double(const wchar_t * str, const wchar_t ** after = nullptr);
      float to_float(const std::string & str, const char ** after = nullptr);
      float to_float(const std::wstring & str, const wchar_t ** after = nullptr);
      double to_double(const std::string & str, const char ** after = nullptr);
      double to_double(const std::wstring & str, const wchar_t ** after = nullptr);
      long double to_long_double(const std::string & str, const char ** after = nullptr);
      long double to_long_double(const std::wstring & str, const wchar_t ** after = nullptr);
      std::string to_str(float val);
      std::wstring to_wstr(float val);
      std::string to_str(double val);
      std::wstring to_wstr(double val);
      std::string to_str(long double val);
      std::wstring to_wstr(long double val);
2021-06-23 17:01:43 +02:00
tomasz.sowa c1f1dc96df added Space::serialize_to_string(StreamType & stream) template 2021-06-22 17:52:55 +02:00
tomasz.sowa 99fbdc1635 in Log::~Log(): removed call to save_log_and_clear()
it creates a problem if a buffer is destroyed first:
2021-06-20 18:19:53 +02:00
tomasz.sowa 4a2a99a77d removed a comment 2021-06-20 17:39:37 +02:00
tomasz.sowa 0865c41e48 make depend 2021-06-20 16:47:12 +02:00